Systems and methods for adjusting light emitted from a display

ABSTRACT

Systems and methods for adjusting light emitted from a display of a device are provided. The adjusting includes obtaining, from light of an environment detected by at least one sensor, a measured color of light of the environment, and obtaining, from light of the environment detected by at least one sensor, a measured brightness of light of the environment. In response to the obtaining the measured color and the measured brightness of light, a color of light emitted from the display is adjusted from an initial color prior to the adjusting to a target color that matches the measured color. Further, a brightness of light emitted from the display is adjusted from an initial brightness emitted by the display prior to the adjusting to a target brightness that matches the measured brightness of light.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Application Ser. No.17,282,101, filed Apr. 1, 2021, which claims the benefit ofInternational Application No. PCT/US2018/054867, filed Oct. 8, 2018,which are hereby incorporated by reference in their entirety for allpurposes.

TECHNICAL FIELD

This application relates generally to electronic devices, including butnot limited to adjusting light emitted from a display of the devices.

BACKGROUND

Electronic devices are extraordinarily popular and are typically foundin every room of a house. While these devices are devised to augment thelives of their users, the digital displays of the devices often become anuisance to users. Users have complained that the digital displays ofthe electronic devices either act as a black mirror when disabled or aglowing beacon from light emitted by a backlight of the display.Moreover, these displays do not have a granularity of brightness that isneeded to have the device replicate and blend into a surroundingenvironment. Blending a digital display into a physical environment toreplicate a physical picture or painting is difficult, since physicalobjects reflect light while digital objects emit light.

Previous attempts to render these digital digitals and blend them into asurrounding physical environment, such as conventional digital pictureframe devices, have been unsuccessful. For instance, conventionaldigital displays often have a granularity to adjust a brightness of thedisplay by intervals of 10 percent, or 5 percent, which is not fineenough to produce an image that replicates a physical rendering of thedisplay. Setting a display to an all-black image does not prevent thedisplay from emitting light, as a backlight of the display can bleedthrough a periphery of the display.

Additionally, these displays often do not account for a color of lightemitted throughout the day. The color of an environment differsthroughout a day. For example, in direct sunlight at midday, visiblelight hues appear more blue and green (e.g., emissions in a 400nanometer to 550 nanometer wavelength band are dominate) than at dusk,which may cause a person to become more alert and aware due to theassociate of these colors with sunlight. Moreover, at sunset, huesappear orange and red (colors in the 600 nanometer to 700 nanometerrange are dominate) because shorter wavelengths of visible light arerefracted by the atmosphere, which a person then associates withsleepfulness.

These distractions lead to the conventional digital picture frame devicebeing turned off or even discarded, since the conventional digitalpicture frame device is no longer augmenting, but instead distractingfrom, its surrounding environment.

Given the prevalence of digital displays, it is beneficial to adjustlight emitted by these displays to blend in with a physical environmentand replicate a physical object without human interaction.

SUMMARY

The present disclosure addresses the above-identified shortcomings byproviding electronic devices and methods that are applicable in a homeor office environment to provide a distraction-free interface thatadjusts light emitted from a display of device to blend into theenvironment. The electronic device is configured to provide variousmodes of display according to different characteristics of light thatare detected by the device and a state of the device.

In accordance with one aspect of the present disclosure, a method ofadjusting light emitted from a display of a device is provided at thedevice. The device includes one or more processors, memory storing oneor more programs for execution by the one or more processors, thedisplay, and one or more sensors. The one or more programs singularly orcollectively include obtaining, from light of a surrounding environmentdetected by at least one sensor in the one or more sensors, a measuredcolor of light of the surrounding environment. The one or more programsfurther include obtaining, from light of the surrounding environmentdetected by at least one sensor in the one or more sensors, a measuredbrightness of light of the surrounding environment. The one or moreprograms further include adjusting, in response to the obtaining themeasured color and the measured brightness of light, a color of lightemitted from the display from an initial color of light emitted by thedisplay prior to the adjusting to a target color of light that matchesthe measured color of light, and a brightness of light emitted from thedisplay from an initial brightness of light emitted by the display priorto the adjusting to a target brightness of light that matches themeasured brightness of light.

In some embodiments, the at least one sensor includes a red-green-bluecolor sensor that is configured to detect the color of light of thesurrounding environment, and an ambient light sensor that is configuredto detect the brightness of light of the surrounding environment.

In some embodiments, the measured color of light is quantified at one ormore wavelengths or wavelength bands of light in the visible spectrum.

In some embodiments, the measured color of light is quantified as acolor temperature.

In some embodiments, the measured brightness of light is quantified on abrightness scale or a perceived brightness scale.

In some embodiments, the adjusting the brightness of light furtherincludes referencing a lookup table that is accessible to the device.The lookup table includes a first data field that includes one or morevalues of color of light, and a second data field that includes one ormore values of brightness of light. Each value of brightness of light ofthe second data field corresponds to at least one value of color oflight of the first data field. Using a correspondence between a color oflight in the first data field and a brightness of line in the seconddata field to determine the target color or the target brightness.

In some embodiments, the obtaining the measured color of light and theobtaining the measured brightness of light is concurrently conducted.

In some embodiments, the obtaining the measured color of light and theobtaining of the measured brightness of light are conducted on arecurring basis.

In some embodiments, the recurring basis is one second time intervals.

In some embodiments, the target color of light is exactly equivalent tothe measured color of light.

In some embodiments, the target brightness of light is a brightness oflight that is a user perceived equivalence of the measured brightness oflight.

In some embodiments, the user perceived equivalence of the measuredbrightness of light is based on an offset brightness relative to themeasured brightness.

In some embodiments, the offset brightness is between 0.1% and 10% ofthe measured brightness.

In some embodiments, the adjusting the color of light emitted from thedisplay and the adjusting the brightness of light emitted from thedisplay are implemented as a transition between an (i) initial color oflight emitted from the display prior to the adjusting and (ii) thetarget color and between (i) an initial brightness of light emitted fromthe display prior to the adjusting and (ii) the target brightness,wherein the transition occurs over a predetermined period of time.

In some embodiments, the triggering, in accordance with a determinationthat the measured brightness of light satisfies a first thresholdbrightness value, a device state.

In some embodiments, the device state disables the display.

In some embodiments, the device state displays predetermined informationon the display.

In some embodiments, the method further includes removing, in accordancewith a determination that the measured brightness satisfies a secondthreshold brightness value, the device from the device state. The secondthreshold brightness value is greater than the first thresholdbrightness value.

In some embodiments, the device further includes a microphone and one ormore speakers. The one or more programs further include outputting apulse of sound through the one or more speakers, and receiving,responsive to the outputting, the pulse of sound through the microphone.The one or more programs also include determining, responsive to thereceiving of the pulse of sound, if one or more users of the device islocated in the surrounding environment.

In some embodiments, the pulse of sound is inaudible.

In some embodiments, a user of the device overrides the adjusting tocause the display to emit a brightness of light specified by the user.

In some embodiments, the one or more programs include determining when auser is engaged with the device. In accordance with a determination thatthe device is currently engaged, triggering a first state of the device,the first state of the device associated with a first brightness oflight emitted from the display of the device. The first brightnessexceeds the target brightness. In accordance with a determination thatthe device is currently unengaged, triggering the device to return toemitting light at the target brightness.

In some embodiments, the engagement with the device includes a vocalinteraction, received through a microphone of the device, a touchinteraction, received through the display of the device, wherein thedisplay is a touch sensitive display, or an auxiliary interaction. Theauxiliary interaction is either received from a remote computer system,or provided through one or more programs of the device.

In some embodiments, the, determination that the device is currentlyunengaged occurs in accordance with a determination that a previouslyreceived engagement with the device satisfies a threshold period oftime.

In some embodiments, the device further incudes a camera. The one ormore programs including in accordance with a determination that at leastone of the measured color and the measured brightness of light satisfiesa threshold value of confidence a determining, from measured lightcaptured by the camera, the color of light of the surroundingenvironment determining, from measured light captured by the camera, thebrightness of light of the surrounding environment, using the determinedcolor of light of the surrounding environment as detected by the one ormore sensors and as captured by the camera, to specify the target colorof light, and using the determined brightness of light of thesurrounding environment as detected by the one or more sensors and ascaptured by the camera, to specify the target brightness of light.

In accordance with various embodiments of this application, the deviceadjusts the brightness and color of light emitted according to adetected brightness and color of the surrounding environment. Theadjusted brightness and color of the display are based on the detectedbrightness and color of the environment, a user setting, a type ofinformation being displayed by the device, or a combination thereof.Accordingly, the device adjusts the brightness of the display and thecolor of the display in such a wide range and with such granularity thatthe display is able to replicate a physical photo or painting whileblending in to the surrounding environment.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described implementations,reference should be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is an example smart home environment in accordance an embodimentof the present disclosure;

FIG. 2A is an example operating environment in which a voice-activatedelectronic device (e.g., a display assistant device) interacts with acast device, a client device or a server system of a smart homeenvironment in accordance with an embodiment of the present disclosure;

FIG. 2B is another example operating environment in whichvoice-activated electronic devices interact with cast devices, clientdevices or a server system of a smart home environment in accordancewith an embodiment of the present disclosure;

FIG. 3 is a block diagram illustrating an example display assistantdevice that is applied as a voice interface to collect user voicecommands in a smart home environment in accordance with an embodiment ofthe present disclosure;

FIG. 4 is a block diagram illustrating an example server in the serversystem 140 of a smart home environment in accordance with an embodimentof the present disclosure;

FIGS. 5A, 5B, 5C, 5D, 5E, and 5F are collectively illustrate perspectiveview, a front view, a rear view, a side view, a top view and a bottomview of a display assistant device in accordance with an embodiment ofthe present disclosure;

FIGS. 6A, 6B, 6C, and 6D collectively illustrate a flow chart of amethod of adjusting light emitted from a display assistant device inaccordance with an embodiment of the present disclosure, in whichoptional portions are illustrated using a dashed-line or dashed-box; and

FIGS. 7A and 7B collectively illustrate a user interface for managing adisplay of a display assistant device in accordance with an embodimentof the present disclosure.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DESCRIPTION OF IMPLEMENTATIONS

While digital displays have revolutionized how information is presentedto people, the displays often detract from their surroundingenvironments. For instance, these displays often adjust a brightness oflight according to only a detected brightness of light in order toincrease a readability of the display. However, this adjustment does notallow for the device to blend into the surrounding environment morenaturally, nor does it allow for the device to replicate an actualpicture or painting since readability is typically associated with abrighter display. Additionally, digital displays often do not adjust thelight emitted therefrom to account for a color of light of thesurrounding environment. While the displays reduce particular wavelengthbands at certain times (e.g., a blue light filter applied at nighttime), they are not capable of replicating the full spectrum of colorsthat are visible through the day in a dynamic environment such as a homeor office, where light conditions vary greatly throughout each day.

In accordance with some implementations of the invention, an electronicdevice includes a screen configured to provide additional visualinformation (e.g., display media content) to a user of the device. Thelight emitted by the display is quantified by a brightness of light anda color of light. Accordingly, light from a surrounding environment ofthe device is detected using one or more sensors coupled to the device.These sensors capture a brightness and a color of light in theenvironment, which are used to determine the emitted light of thedisplay. The display has a fine granularity of possible brightness andcolors that allow the device to adapt to a wide variety of differentenvironment conditions, leading to a more natural looking display thatblends into the surrounding environment.

Specifically, Systems and methods for adjusting light emitted from adisplay of a device are provided. The adjusting includes obtaining, fromlight of an environment detected by at least one sensor, a measuredcolor of light of the environment, and obtaining, from light of theenvironment detected by at least one sensor, a measured brightness oflight of the environment. In response to the obtaining the measuredcolor and the measured brightness of light, a color of light emittedfrom the display is adjusted from an initial color prior to theadjusting to a target color that matches the measured color. Further, abrightness of light emitted from the display is adjusted from an initialbrightness emitted by the display prior to the adjusting to a targetbrightness that matches the measured brightness of light.

Reference will now be made in detail to implementations, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the various describedimplementations. However, it will be apparent to one of ordinary skillin the art that the various described implementations may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, circuits, and networks have not been describedin detail so as not to unnecessarily obscure aspects of theimplementations.

FIG. 1 is an example smart home environment 100 in accordance with someimplementations. The smart home environment 100 includes a structure 150(e.g., a house, office building, garage, or mobile home) with variousintegrated devices. It will be appreciated that devices also beintegrated into a smart home environment 100 that does not include anentire structure 150, such as an apartment, condominium, or officespace. The depicted structure 150 includes a plurality of rooms 152,separated at least partly from each other via walls 154. In someembodiments, the walls 154 are interior walls or exterior walls. In someimplementations, each room further include a floor 156 and a ceiling158.

One or more media devices are disposed in the smart home environment 100to provide media content that is stored at a local content source orstreamed from a remote content source (e.g., content host(s) 114). Themedia devices can be classified to two categories: media output devices106 that directly output the media content to audience, and cast devices108 that are networked to stream media content to the media outputdevices 106. Examples of the media output devices 106 include, but arenot limited to television (TV) display devices and music players.Examples of the cast devices 108 include, but are not limited to,set-top boxes (STBs), DVD players and TV boxes. In the example smarthome environment 100, the media output devices 106 are disposed in morethan one location, and each media output device 106 is coupled to arespective cast device 108 or includes an embedded casting unit. Themedia output device 106-1 includes a TV display that is hard wired to aDVD player or a set top box 108-1. The media output device 106-2includes a smart TV device that integrates an embedded casting unit tostream media content for display to its audience. The media outputdevice 106-3 includes a regular TV display that is coupled to a TV box108-3 (e.g., Google TV or Apple TV products), and such a TV box 108-3streams media content received from a media content host server 114 andprovides an access to the Internet for displaying Internet-based contenton the media output device 106-3.

In addition to the media devices 106 and 108, one or more electronicdevices 190 are disposed in the smart home environment 100 to collectaudio inputs for initiating various media play functions of the devices190 and/or media devices 106 and 108. In some implementations, thedevices 100 are configured to provide media content that is storedlocally or streamed from a remote content source. In someimplementations, these voice-activated electronic devices 190 (e.g.,devices 190-1, 190-2 and 190-3) are disposed in proximity to a mediadevice, for example, in the same room with the cast devices 108 and themedia output devices 106. Alternatively, in some implementations, avoice-activated electronic device 190-4 is disposed in a room having oneor more smart home devices but not any media device. Alternatively, insome implementations, a voice-activated electronic device 190 isdisposed in a location having no networked electronic device. Thisallows for the devices 190 to communicate with the media devices andshare content which is being displayed on one device to another device(e.g., from device 190-1 to device 190-2 and/or media devices 108).

The electronic device 190 includes at least one microphones, a speaker,a processor and memory storing at least one program for execution by theprocessor. The speaker is configured to allow the electronic device 190to deliver voice messages to a location where the electronic device 190is located in the smart home environment 100, thereby broadcastinginformation related to a current media content being displayed,reporting a state of audio input processing, having a conversation withor giving instructions to a user of the electronic device 190. Forinstance, in some implementations in response to a user query the deviceprovides audible information to the user through the speaker. As analternative to the voice messages, visual signals could also be used toprovide feedback to the user of the electronic device 190 concerning thestate of audio input processing, such as a notification displayed on thedevice.

In accordance with some implementations, the electronic device 190 is avoice interface device that is network-connected to provide voicerecognition functions with the aid of a cloud cast service server 116and/or a voice/display assistance server 112. For example, theelectronic device 190 includes a smart speaker that provides music(e.g., audio for video content being displayed) to a user and allowseyes-free and hands-free access to voice assistant service (e.g., GoogleAssistant). Optionally, the electronic device 190 is a simple and lowcost voice interface device, e.g., a speaker device and a displayassistant device (including a display screen having no touch detectioncapability).

In some implementations, the voice-activated electronic devices 190includes a display assistant device (e.g., 190-2 and 190-4) thatintegrates a display screen in addition to the microphones, speaker,processor and memory. The display screen is configured to provideadditional visual information (e.g., media content, informationpertaining to media content, etc.) in addition to audio information thatcan be broadcast via the speaker of the voice-activated electronicdevice 190. In some implementations, when a user is nearby and his orher line of sight is not obscured, the user reviews the additionalvisual information directly on the display screen of the displayassistant device. Optionally, the additional visual information providesfeedback to the user of the electronic device 190 concerning the stateof audio input processing. Optionally, the additional visual informationis provided in response to the user's previous voice inputs (e.g., userqueries), and is related to the audio information broadcast by thespeaker. In some implementations, the display screen of thevoice-activated electronic devices 190 includes a touch display screenconfigured to detect touch inputs on its surface (e.g., instructionsprovided through the touch display screen). Alternatively, in someimplementations, the display screen of the voice-activated electronicdevices 190 is not a touch display screen, which is relatively expensiveand can compromise the goal of offering the display assistant device 190as a low cost user interface solution.

When voice inputs from the electronic device 190 are used to control theelectronic device 190 and/or media output devices 106 via the castdevices 108, the electronic device 190 effectively enables a new levelof control of cast-enabled media devices independently of whether theelectronic device 190 has its own display. In an example, the electronicdevice 190 includes a casual enjoyment speaker with far-field voiceaccess and functions as a voice interface device for Google Assistant.The electronic device 190 could be disposed in any room in the smarthome environment 100. When multiple electronic devices 190 aredistributed in multiple rooms, they become audio receivers that aresynchronized to provide voice inputs from all these rooms. For instant,in some implementations a first electronic device 190 receives a userinstruction that is directed towards a second electronic device 190-2(e.g., a user instruction of “OK Google, show this photo album on theKitchen device.”).

Specifically, in some implementations, the electronic device 190includes a Wi-Fi speaker with a microphone that is connected to avoice-activated personal assistant service (e.g., Google Assistant). Auser could issue a media play request via the microphone of electronicdevice 190, and ask the personal assistant service to play media contenton the electronic device 190 itself and/or on another connected mediaoutput device 106. For example, the user could issue a media playrequest by saying to the WiFi speaker “OK Google, Only show photos of mycats for the next two hours on all connected devices.” The personalassistant service then fulfils the media play request by playing therequested media content on the requested devices using a default ordesignated media application.

A user could also make a voice request via the microphone of theelectronic device 190 concerning the media content that has already beenplayed and/or is being played on a display device. For instance, in someimplementations a user instructs the device to provide informationrelated to a current media content being displayed, such as ownershipinformation or subject matter of the media content. In someimplementations, closed captions of the currently displayed mediacontent are initiated or deactivated on the display device by voice whenthere is no remote control or a second screen device is available to theuser. Thus, the user can turn on the closed captions on a display devicevia an eyes-free and hands-free voice-activated electronic device 190without involving any other device having a physical user interface, andsuch a voice-activated electronic device 190 satisfies federalaccessibility requirements for users having hearing disability. In someimplementations, a user may want to take a current media session withthem as they move through the house. This requires the personalassistant service to transfer the current media session from a firstcast device to a second cast device that is not directly connected tothe first cast device or has no knowledge of the existence of the firstcast device. Subsequent to the media content transfer, a second outputdevice 106 coupled to the second cast device 108 continues to play themedia content previously a first output device 106 coupled to the firstcast device 108 from the exact point within a photo album or a videoclip where play of the media content was forgone on the first outputdevice 106.

In some implementations, the voice-activated electronic devices 190,smart home devices could also be mounted on, integrated with and/orsupported by a wall 154, floor 156 or ceiling 158 of the smart homeenvironment 100 (which is also broadly called as a smart homeenvironment in view of the existence of the smart home devices).

In some embodiments, the smart home devices in the smart homeenvironment 100 includes one or more intelligent, multi-sensing,network-connected camera systems 132. In some implementations, contentthat is captured by the camera systems 132 be displayed on theelectronic devices 190 at a request of a user (e.g., a user instructionof “OK Google, Show the baby room monitor.”) and/or according tosettings of the home environment 100 (e.g., a setting to display contentcaptured by the camera systems during the evening or in response todetecting an intruder).

In some implementations, each of the voice-activated electronic devices190 is capable of data communications and information sharing with othervoice-activated electronic devices 190, a central server orcloud-computing system 140, and/or other devices that arenetwork-connected. In some implementations, data communications arecarried out using any of a variety of custom or standard wirelessprotocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave,Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and/or any of avariety of custom or standard wired protocols (e.g., Ethernet, HomePlug,etc.), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

In some implementations, the electronic devices 190 serve as wireless orwired repeaters. In some implementations, the electronic devices 190further communicate with each other via a connection (e.g., networkinterface 160) to a network, such as the Internet 110. Through theInternet 110, the cast devices 108, and the electronic devices 190communicate with a smart server system 140 (also called a central serversystem and/or a cloud-computing system herein). Optionally, the smartserver system 140 is associated with a manufacturer, support entity, orservice provider associated with the cast devices 108 and the mediacontent displayed to the user.

Accordingly, in some implementations the smart server system 140includes a voice/display assistance server 112 that processes audioinputs collected by voice-activated electronic devices 190, one or morecontent hosts 114 (e.g., databases of a cloud server) that provide themedia content, a cloud cast service server 116 creating a virtual userdomain based on distributed device terminals, and a device registry 118that keeps a record of the distributed device terminals in the virtualuser environment. Examples of the distributed device terminals include,but are not limited to the voice-activated electronic devices 190, castdevices 108, media output devices 106 and smart home devices (e.g.,camera system 132). In some implementations, these distributed deviceterminals are linked to a user account (e.g., a Google user account) inthe virtual user domain. Furthermore, in some implementations eachdisplay device 190 is linked to one or more user accounts. Accordingly,in some implementation a device 190 has access to display media contentthat is associated with the user accounts of the device (e.g., photoalbums of the user accounts).

In some implementations, the network interface 160 includes aconventional network device (e.g., a router). The smart home environment100 of FIG. 1 further includes a hub device 180 that is communicativelycoupled to the network(s) 110 directly or via the network interface 160.The hub device 180 is further communicatively coupled to one or more ofthe above intelligent, multi-sensing, network-connected devices (e.g.,the cast devices 108, the electronic devices 190, the smart home devicesand the client device 104). Each of these network-connected devicesoptionally communicates with the hub device 180 using one or more radiocommunication networks available at least in the smart home environment100 (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi and other radiocommunication networks). In some implementations, the hub device 180 anddevices coupled with/to the hub device can be controlled and/orinteracted with via an application running on a smart phone, householdcontroller, laptop, tablet computer, game console or similar electronicdevice. In some implementations, a user of such controller applicationcan view status of the hub device or coupled network-connected devices,configure the hub device to interoperate with devices newly introducedto the home network, commission new devices, and adjust or view settingsof connected devices, etc.

FIG. 2A is an example operating environment 200 in which avoice-activated electronic device 190 (e.g., a display assistant device)interacts with a cast device 108, a client device 104 or a server system140 of a smart home environment 100 in accordance with someimplementations of the present disclosure. The voice-activatedelectronic device 190 is configured to receive audio inputs from anenvironment in proximity to the voice-activated electronic device 190(e.g., a user of the device). Optionally, the electronic device 190stores the audio inputs and at least partially processes the audioinputs locally (e.g., processes a user query). Optionally, theelectronic device 190 transmits the received audio inputs or thepartially processed audio inputs to a voice/display assistance server112 via the communication networks 110 for further processing. The castdevice 108 is configured to obtain media content or Internet contentfrom one or more content hosts 114 for display on an output device 106coupled to the cast device 108. As explained above, the cast device 108and the voice-activated electronic device 190 are linked to each otherin a user domain, and more specifically, associated with each other viaa user account in the user domain. Information of the cast device 108and information of the electronic device 190 are stored in the deviceregistry 118 in association with the user account.

In some implementations, one or more of the contents hosts 114 is adatabase which stores media files associated with a plurality of users.Similarly, in some implementations one or more of the content hosts 114is a media content provide which provides media content. In someimplementations, the media content provided by the content hosts in suchimplementations is private media content (e.g., media content that islicensed from the content provider such as works of art), or pubic mediacontent (e.g., a library of generic stock media).

In some implementations, the cast device 108 does not include anydisplay screen, and the voice-activated electronic device 190 includes adisplay assistant device that has a display screen. Both the cast device108 and the display assistant device 190 have to rely on the clientdevice 104 to provide a user interface during a commissioning process.Specifically, the client device 104 is installed with an applicationthat enables a user interface to facilitate commissioning of a new castdevice 108 or a new display assistant device 190 disposed in proximityto the client device 104. In some implementations, a user sends arequest on the user interface of the client device 104 to initiate acommissioning process for the new cast device 108 or display assistantdevice 190 that needs to be commissioned. After receiving thecommissioning request, the client device 104 establishes a short rangecommunication link with the new cast device 108 or display assistantdevice 190 that needs to be commissioned. Optionally, the short rangecommunication link is established based near field communication (NFC),Bluetooth, Bluetooth Low Energy (BLE) and the like. The client device104 then conveys wireless configuration data associated with a wirelesslocal area network (WLAN) to the new cast device 108 or displayassistant device 190. The wireless configuration data includes at leasta WLAN security code (i.e., service set identifier (SSID) password), andoptionally includes an SSID, an Internet protocol (IP) address, proxyconfiguration and gateway configuration. After receiving the wirelessconfiguration data via the short range communication link, the new castdevice 108 or display assistant device 190 decodes and recovers thewireless configuration data, and joins the WLAN based on the wirelessconfiguration data.

Additional user domain information is entered on the user interfacedisplayed on the client device 104, and used to link the new cast device108 or display assistant device 190 to an account in a user domain.Optionally, the additional user domain information is conveyed to thenew cast device 108 or display assistant device 190 in conjunction withthe wireless communication data via the short range communication link.Optionally, the additional user domain information is conveyed to thenew cast device 108 or display assistant device 190 via the WLAN afterthe new device has joined the WLAN.

Once the cast device 108 and display assistant device 190 have beencommissioned into the user domain, the cast device 108, the outputdevice 106 and their associated media play activities could becontrolled via two control paths (control path A and control path B). Inaccordance with control path A, a cast device application or one or moremedia play applications installed on the client device 104 are used tocontrol the cast device 108 and its associated media play activities.Alternatively, in accordance with control path B, the display assistantdevice 190 is used to enable eyes-free and hands-free control of thecast device 108 and its associated media play activities (e.g., playbackof media content play on the output device 106), as well as to displaymedia on the device 190 itself.

In some implementations, the cast device 108 and display assistantdevice 190 are two distinct and different devices that are configured toact as a cast receiver device and a cast transmitter device,respectively. The display assistant device 190 can provide informationor content (which is generated locally or received from another source)to be projected onto the output device 106 via the cast device 108.Alternatively, in some implementations, the cast device 108 and displayassistant device 190 are combined in an integrated cast device that iscoupled to the output display assist device 106.

In some implementations, the smart home environment 100 includes one ormore smart home devices 120 (e.g., camera systems 132 in FIG. 1 ).Regardless of whether a smart home device 120 has a display screen, itcan rely on the client device 104 and/or display assist device 190 toprovide a user interface during a commissioning process. Specifically,the client device 104 is installed with a smart device application thatenables a user interface to facilitate commissioning of a new smart homedevice 120. Like a new cast device 108 or display assistant device 190,the new smart home device 120 can establish a short range communicationlink with the client device 104, and the wireless configuration data arecommunicated to the new smart home device 120 via the short rangecommunication link, allowing the smart home device 120 to join the WLANbased on the wireless configuration data. Further, the smart home device120 is optionally linked to the account of the user domain to which thecast device 108 and display assistant device 190 are linked as well.Once the smart home device 120 and the display assistant device 190 havebeen commissioned into the user domain, the smart home device 120 couldbe monitored and controlled via the display assistant device 190 inaccordance with Control Path C as the cast device 108 is controlled viathe display assistant device 190 in accordance with Control Path B. Forexample, voice commands can be inputted into the display assistantdevice 190 to review recording of an outdoor camera 132 mounted next toa door and control a door lock based on security events detected in therecordings.

Referring to FIG. 2A, after the cast device 108 and the voice-activatedelectronic device 190 are both commissioned and linked to a common userdomain, the voice-activated electronic device 190 can be used as a voiceuser interface to enable eyes-free and hands-free control of mediacontent streaming to the cast device 108 involving no remote control,client device 104, other electronic devices 190, and/or other secondscreen device. For example, in some implementations the user gives voicecommands such as “Show photos of Morgan and I on the Kitchen display.”Accordingly, a photo or video clip is streamed to a cast device 108and/or electronic device 190 associated with the “Kitchen display.” Theclient device 104 is not involved, nor is any cast device application ormedia play application loaded on the client device 104.

The cloud cast service 116 is the proxy service that communicativelylinks the voice-activated electronic device 190 to the cast device 108and makes casting to the cast device 108 possible without involving anyapplications on the client device 104. For example, a voice message isrecorded by an electronic device 190, and the voice message isconfigured to request media play on a media output device 106.Optionally, the electronic device 190 partially processes the voicemessage locally. Optionally, the electronic device 190 transmits thevoice message or the partially processed voice message to avoice/display assistance server 112 via the communication networks 110for further processing. A cloud cast service server 116 determines thatthe voice message includes a first media play request (e.g., a userquery for media content), and that the first media play request includesa user voice command to play media content on a media output device 106and/or an electronic device 190 and a user voice designation of themedia output device 106 and/or the electronic device 190. The user voicecommand further includes at least information the media content (e.g.,photos and/or videos that include Morgana as a subject matter) thatneeds to be played. Furthermore, in some implementations the user voicecommand further includes an instruction for the electronic device 190 toimplement, such as a modification to a particular media content or toshare media content with another user.

In accordance with the voice designation of the media output device, thecloud cast service server 116 in a device registry 118 a cast deviceassociated in the user domain with the electronic device 190 and coupledto the media output device 106. The cast device 108 is configured toexecute one or more media play applications for controlling the mediaoutput device 106 to play media content received from one or more mediacontent hosts 114. Then, the cloud cast service server 116 sends to thecast device 108 a second media play request including the information ofthe first media play application and the media content that needs to beplayed. Upon receiving the information sent by the cloud cast serviceserver 116, the cast device 108 executes the first media playapplication and controls the media output device 106 to play therequested media content.

In some implementations, the user voice designation of the media outputdevice 106 and/or an electronic device 190 includes description of thedestination media output device and/or electronic device. The cloud castservice server 116 identifies in the registry the destination mediaoutput device and/or the electronic device 190 among a plurality ofmedia output devices and/or a plurality of electronic device 190according to the description of the destination media output deviceand/or the electronic device. In some implementations, the descriptionof the destination media output device and/or the electronic deviceincludes at least a brand (“Samsung TV”) or a location of the mediaoutput device 106 and/or the electronic device 190 (“my Living Roomdevice”).

FIG. 2B is another example operating environment 250 in whichvoice-activated electronic devices 190 interact with cast devices 106,client devices 104, other electronic devices 190, and/or a server system140 of a smart home environment 100 in accordance with someimplementations of the present disclosure. The smart home environment100 includes a first cast device 108-1 and a first output device 106-1,or a first electronic device 190-1, coupled to the first cast device108-1. The smart home environment 100 also includes a second cast device108-2 and a second output device 106-2, or second electronic device190-2, coupled to the second cast device 108-2. The cast devices 108-1and 108-2, or electronic devices 190-1 and 190-2, are optionally locatedin the same location (e.g., the living room) or two distinct locations(e.g., two rooms) in the smart home environment 100. Each of the castdevices 108-1 and 108-2, or the electronic devices 190, is configured toobtain media or Internet content from media hosts 114 for display on theoutput device 106 coupled to the respective cast device 108-1 or 108-2and/or the electronic devices 190. Both the first and second castdevices, or the electronic devices 190, are communicatively coupled tothe cloud cast service server 116 and the content hosts 114.

The smart home environment 100 further includes one or morevoice-activated electronic devices 190 that are communicatively coupledto the cloud cast service server 116 and the voice/display assistanceserver 112. The one or more voice-activated electronic devices 190includes at least one display assistant device (e.g., display assistantdevice 190-2). In some implementations, the voice-activated electronicdevices 190 are disposed independently of the cast devices 108 and theoutput devices 106. For example, as shown in FIG. 1 , the electronicdevice 190-4 is disposed in a room where no cast device 108 or outputdevice 106 is located. In some implementations, the first electronicdevice 190-1 is disposed in proximity to the first cast device 108-1 andthe first output device 106-1, e.g., the first electronic device 190-1,the first cast device 108-1 and the first output device 106-1 arelocated in the same room. Optionally, the second electronic device 190-2is disposed independently of or in proximity to the second cast device108-2 and the second output device 106-2.

When media content is being played on the first output device 106-1 orthe electronic device 190, a user may send a voice command to any of theelectronic devices 190 (e.g., 190-1 or 190-2 in FIG. 2B) to request playof the media content to be transferred to the second output device 106-2or a second electronic device 190-2. The voice command includes a mediaplay transfer request (e.g., a user instruction to transfer the mediacontent). The voice command is transmitted to the cloud cast serviceserver 116. The cloud cast service server 116 sends a media displayinformation request to the first cast device 108-1 to request instantmedia play information of the media content that is currently beingplayed on the first output device 106-1 coupled to the first cast device108-1. The first cast device 108-1 then returns to the cloud castservice server 116 the requested instant play information including atleast information the media content that is currently being played(e.g., “Lady Gaga—Super Bowl 2016”), and a temporal position related toplaying of the media content. The second cast device 108-2 or the secondelectronic device 190-2 then receives a media display request includingthe instant play information from the cloud cast service server 116, andin accordance with the instant play information, executes the firstmedia play application that controls the second output device 106-2 orthe second electronic device 190-2 to play the media content from thetemporal location.

FIG. 3 is a block diagram illustrating an example display assistantdevice 300 that is applied as a voice interface to collect user voicecommands in a smart home environment 100 and/or display media content inaccordance with some implementations. The display assistant device 300typically includes one or more processing units (CPUs) 302, one or morenetwork interfaces 304, memory 306, and one or more communication buses308 for interconnecting these components (sometimes called a chipset).The display assistant device 300 includes one or more output devices312, including one or more speakers 350 and a display 352. The displayassistant device 300 also includes one or more input devices 310 thatfacilitate user input, including one or more microphones 342, a volumecontrol 344 and a privacy control 346. The volume control 346 isconfigured to receive a user action (e.g., a press on a volume up buttonor a volume down button, a press on both volumes up and down buttons foran extended length of time) that controls a volume level of the speakers350 or resets the display assistant device 300. The privacy control 346is configured to receive a user action that controls privacy settings ofthe display assistant device (e.g., whether to deactivate themicrophones 342). In some implementations, the input devices 310 of thedisplay assistant device 300 include a touch detection module (not shownin FIG. 3 ) that is integrated on the display panel 352 and configuredto detect touch inputs on its surface. In some implementations, theinput devices 310 of the display assistant device 300 include a cameramodule configured to capture a video stream and/or a picture of a fieldof view. For instance, in some implementations a user may instruct thedisplay assistant device 300 to show photos of one or more objects(e.g., people and/or animals) that are in a field of view of the device.Accordingly, the display device 300 and/or an external server (e.g.,content host 114 and/or voice/display assistance server 112) to detectthe faces in the field of view. Content that is stored on the displaydevice 300 and/or the content host 114 is analyzed to determined mediacontent that includes the detected faces. This media content is thendisplayed on the requested device (e.g., a device the request originatedfrom). Alternatively, in some implementations, the input devices 310 ofthe display assistant device 300 does not include any camera or touchdetection module, because they relatively expensive and can compromisethe goal of offering the display assistant device 300 as a low cost userinterface solution.

In some implementations, the display assistant device 300 furtherincludes a presence sensor 360 configured to detect a presence of a userin a predetermined area surrounding the display assistant device 300.Under some circumstances, the display assistant device 300 operates at asleep or hibernation mode that deactivates detection and processing ofaudio inputs, and does not wake up from the sleep or hibernation mode orlisten to the ambient (i.e., processing audio signals collected from theambient) until the presence sensor 360 detects a presence of a user inthe predetermined area. An example of the presence sensor 360 is anultrasonic sensor configured to detect a presence of a user. Forinstance, in some implementations the display device 300 is configuredto sleep or hibernate when a presence of a user is not detected toconserve energy consumption of the device.

Memory 306 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. Memory 306, optionally, includes one or more storagedevices remotely located from one or more processing units 302. Memory306, or alternatively the non-volatile memory within memory 306,includes a non-transitory computer readable storage medium. In someimplementations, memory 306, or the non-transitory computer readablestorage medium of memory 306, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   Operating system 316 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 318 for connecting the display        assistant device 300 to other devices (e.g., the server system        140, the cast device 108, the client device 104, the smart home        devices and the other voice-activated electronic device(s) 190)        via one or more network interfaces 304 (wired or wireless) and        one or more networks 110, such as the Internet, other wide area        networks, local area networks, metropolitan area networks, and        so on;    -   Input/output (I/O) control module 320 for receiving inputs via        one or more input devices 310 enabling presentation of        information at the display assistant device 300 via one or more        output devices 312, including:        -   Voice processing module 322 for processing audio inputs or            voice messages collected in an environment surrounding the            display assistant device 300, or preparing the collected            audio inputs or voice messages for processing at a            voice/display assistance server 112 or a cloud cast service            server 118;        -   Display assistant module 324 for displaying additional            visual information including but not limited to a media            content item (e.g., stock photos or videos), social media            messages, weather information, personal pictures and/or            videos, comments associated with the personal pictures            and/or videos, a state of audio input processing, and            readings of smart home devices; and        -   Touch sense module 326 for sensing touch events on a top            surface of the display assistant device 300; and    -   One or more receiver application 328 for responding to user        commands extracted from audio inputs or voice messages collected        in an environment surrounding the display assistant device 300,        including but not limited to, a media play application, an        Internet search application, a social network application and a        smart device application;    -   Display assistant device data 330 storing at least data        associated with the display assistant device 300, including:        -   Display assistant settings 332 for storing information            associated with the display assistant device 300 itself,            including common device settings (e.g., service tier, device            model, storage capacity, processing capabilities,            communication capabilities, user settings, etc.) and            information of a user account 334 in a virtual user domain            to which the display assistant device 300 is linked;        -   Voice control data 336 for storing audio signals, voice            messages, response messages and other data related to voice            interface functions of the display assistant device 300; and        -   Lookup table module 337 for storing data related to detected            values of brightness 338 and data related to detected values            of color 339.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 306, optionally, stores a subset of the modulesand data structures identified above. Furthermore, memory 306,optionally, stores additional modules and data structures not describedabove.

FIG. 4 is a block diagram illustrating an example server in the serversystem 140 of a smart home environment 100 in accordance with someimplementations. An example server is one of a cloud cast service sever116. The server system 140, typically, includes one or more processingunits (CPUs) 402, one or more network interfaces 404, memory 406, andone or more communication buses 408 for interconnecting these components(sometimes called a chipset). The server system 140 could include one ormore input devices 410 that facilitate user input, such as a keyboard, amouse, a voice-command input unit or microphone, a touch screen display,a touch-sensitive input pad, a gesture capturing camera, or other inputbuttons or controls. Furthermore, the server system 140 could use amicrophone and voice recognition or a camera and gesture recognition tosupplement or replace the keyboard. In some implementations, the serversystem 140 includes one or more cameras, scanners, or photo sensor unitsfor capturing images, for example, of graphic series codes printed onthe electronic devices. The server system 140 could also include one ormore output devices 412 that enable presentation of user interfaces anddisplay content, including one or more speakers and/or one or morevisual displays.

Memory 406 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. Memory 406, optionally, includes one or more storagedevices remotely located from one or more processing units 402. Memory406, or alternatively the non-volatile memory within memory 406,includes a non-transitory computer readable storage medium. In someimplementations, memory 406, or the non-transitory computer readablestorage medium of memory 406, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   Operating system 416 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 418 for connecting the server        system 140 to other devices (e.g., various servers in the server        system 140, the client device 104, the cast device 108, and the        smart home devices 120) via one or more network interfaces 404        (wired or wireless) and one or more networks 110, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   User interface module 420 for enabling presentation of        information (e.g., a graphical user interface for presenting        applications, widgets, websites and web pages thereof, and/or        games, audio and/or video content, text, etc.) at the client        device 104;    -   Command execution module 421 for execution on the server side        (e.g., games, social network applications, smart home        applications, and/or other web or non-web based applications for        controlling the client device 104, the cast devices 108, the        voice activated electronic device 190 (e.g., a display assistant        device 300) and the smart home devices 120 and reviewing data        captured by such devices), including one or more of:        -   a cast device application 422 that is executed to provide            server-side functionalities for device provisioning, device            control, and user account management associated with cast            device(s) 108;        -   one or more media player applications 424 that is executed            to provide server-side functionalities for media display and            user account management associated with corresponding media            sources (e.g., content host 114 of FIG. 1 );        -   one or more smart home device applications 426 that is            executed to provide server-side functionalities for device            provisioning, device control, data processing and data            review of corresponding smart home devices 120; and        -   a voice/display assistant application 428 that is executed            to arrange voice processing of a voice message received from            a voice-activated electronic device 190, directly process            the voice message to extract a user voice command and a            designation of a cast device 108 or another voice-activated            electronic device 190, and/or enable a voice-activated            electronic device 190 to play media content (audio or            video); and    -   Server system data 430 storing at least data associated with        automatic control of media display (e.g., in an automatic media        output mode and a follow-up mode), including one or more of:        -   Client device settings 432 for storing information            associated with the client device 104, including common            device settings (e.g., service tier, device model, storage            capacity, processing capabilities, communication            capabilities, user settings, etc.), and information for            automatic media display control;        -   Cast device settings 434 for storing information associated            with user accounts of the cast device application 422,            including one or more of account access information,            information for device settings (e.g., service tier, device            model, storage capacity, processing capabilities,            communication capabilities, etc.), and information for            automatic media display control;        -   Media player application settings 436 for storing            information associated with user accounts of one or more            media player applications 424, including one or more of            account access information, user preferences of media            content types, review history data, and information for            automatic media display control;        -   Smart home device settings 438 for storing information            associated with user accounts of the smart home applications            426, including one or more of account access information,            information for one or more smart home devices 120 (e.g.,            service tier, device model, storage capacity, processing            capabilities, communication capabilities, etc.);        -   Voice assistance data 440 for storing information associated            with user accounts of the voice/display assistant            application 428, including one or more of account access            information, information for one or more display assistant            devices 190 (e.g., service tier, device model, storage            capacity, processing capabilities, communication            capabilities, etc.); and        -   Media assistance data 442 for storing information associated            with each media file as well as processing of the media            files in order to determine a subject matter of the media            files.

When the server system 140 includes a cloud cast service server 116,memory 406, or the non-transitory computer readable storage medium ofmemory 406, stores the following programs, modules, and data structures,or a subset or superset thereof:

-   -   Device registration module 450 for managing the device registry        118 coupled to the cloud cast service server 116;    -   Cloud cast application 460 for relaying a user voice command        identified in a voice message to one or more of the cast        device(s) 180, the electronic device(s) 190 and the smart home        device(s) 120 that are coupled in a cloud cast user domain; and    -   Status reporting module 470 for maintaining the states of the        cast device(s) 180, the electronic device(s) 190 and the smart        home device(s) 120 that are coupled in a cloud cast user domain.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 406, optionally, stores a subset of the modulesand data structures identified above. Furthermore, memory 406,optionally, stores additional modules and data structures not describedabove.

FIGS. 5A-5E are a perspective view, a front view, a rear view, a sideview, a top view and a bottom view of a display assistant device 500 inaccordance with some implementations, respectively. The displayassistant device 500 includes a base 502 and a screen 504. The base 502is configured for sitting on a surface. The screen 504 has a rearsurface 506 at which the screen 504 is supported by the base.

The base 502 acts a speaker box. A speaker is concealed inside the baseand configured to project sound substantially towards the front view ofthe display assistant device.

In some implementations, a bezel area includes one or more microphoneholes 512. One or more microphones 342 are placed behind the microphoneholes 512 and configured to collect sound from the ambient of thedisplay assistant device 500. In some implements, the display assistantdevice 500 further includes a camera opening 520 configured to capture afield of view of the device. For instance, in some implementations mediacontent is displayed on the device that includes a same subject matteras that which is captured by the camera. In some implementations, thecamera is configured to detect a light condition in the smart homeenvironment 100 where the display assistant device 500 sits. In someimplementations, the display assistant device 500 is configure to adjusta brightness level of its screen 504 according to the light condition.The camera is disposed behind the bezel area and exposed to light via atransparent part of the bezel area, e.g., the sensor opening 520.

In some implementations, the bezel area includes one or more microphoneholes 512. One or more microphones 342 are placed behind the microphoneholes 512 and configured to collect sound from the ambient of thedisplay assistant device 500. In some implements, the display assistantdevice 500 further includes a sensor opening 520 configured to access anambient light sensor and/or a RGB color sensor. The ambient light sensoror RGB color sensor is configured to detect a light condition in thesmart home environment 100 where the display assistant device 500 sits.In some implementations, the display assistant device 500 is configureto adjust a brightness level of its screen 504 according to the lightcondition. The ambient light sensor and the RGB color sensor aredisposed behind the bezel area and exposed to light via transparent partof the bezel area, e.g., the sensor opening 520.

Referring to FIG. 5C, the display assistant device 500 further includesa volume control 344, a privacy control 346 and a power adaptorinterface 518. In this example, the volume control button 514 andprivacy control 346 are disposed on the rear surface 506 of the screen504, and the power adaptor interface 518 is disposed on the rear surface502B of the base 502. However, the present disclosure is not limitedthereto.

In this implementation, the power adaptor interface 518 of the displayassistant device 500 is disposed on the rear surface 502B of the base502. The power adaptor interface 518 includes a male connector toreceive a female connector configured to connect the display assistantdevice 500 to an external power source (e.g., a direct current powersource). In some implementations, the display assistant device 500 hasto be constantly connected to an external power source, and is poweredoff when the external power source is disconnected. Alternatively, insome implementations, the power management unit includes a rechargeablebattery. The rechargeable battery is configured to be charged with theexternal power source, and drive the display assistant device 500temporarily when the external power source is disconnected from thedisplay assistant device 500.

It is noted that FIGS. 5A through 5D are focused on an overall look andmechanical features of the display assistant device 500. More details onfunctions of the display assistant device 500 are described above withreference to FIGS. 1-4 .

FIG. 6 is a flow chart of a method 600 for displaying media files at adevice (e.g., display assist device 300 of FIG. 3 ), in accordance withsome implementations of the present disclosure. The method 600 isperformed at a device, such as display assist device 190 of FIG. 1 . Forexample, instructions for performing the method 600 are stored in thememory 306 and executed by the processor(s) 302 of display device 300.In some implementations, one or more operations described with regard tomethod 600 are performed by server system 140 and/or display device 300.For example, instructions for performing the method 600 are stored inthe memory 406 and executed by the processor(s) 402 of the server system140.

Blocks 602 and 604. Referring to block 502 of FIG. 6A, in accordancewith one aspect of the present disclosure, a method of adjusting lightemitted from a display of a device (e.g., display assistant device 300of FIG. 3 ) is provided. The device includes one or more processors,memory storing one or more programs for execution by the one or moreprocessors, the display (e.g., display 352 of FIG. 3 ), and one or moresensors (e.g., ambient sensor 362, RGB sensor 364, and/or camera 348 ofFIG. 3 ). The one or more programs singularly or collectively includeobtaining, from light of a surrounding environment detected by at leastone sensor in the one or more sensors, a measured color of light of thesurrounding environment.

Block 604. Referring to block 604, in some implementations the at leastone sensor includes an RGB sensor (e.g., RGB sensor 364 of FIG. 3 ).This RGB sensor detects a color of light of a surrounding environmentusing a scale of red, green, and blue primary color combinations. Insome implementations, this scale of red, green, and blue includes usinga color temperate scale, which will be described in more detail withreference to blocks 608 and 610 of FIG. 6A. Moreover, in someimplementations an ambient light sensor (e.g., ambient sensor 362 ofFIG. 3 ) is configured to detect a brightness of light of thesurrounding environment. Furthermore, in some implementations the RGBsensor and the ambient light sensor are subsumed as a single sensor,such as a single-pixel camera (e.g., camera 348 of FIG. 3 ). Additionaldetails and information regarding the camera and sensors of the devicewill be described in more detail below, with reference to at least block658 of FIG. 6D.

Block 606. Referring to block 606, in some implementations the one ormore programs further include obtaining a measured brightness of lightof the surrounding environment. In some implementations, this measuredbrightness of light includes obtaining light of the surroundingenvironment detected by at least one sensor (e.g., ambient light sensor362 and/or camera 348 of FIG. 3 ) in the one or more sensors of thedisplay assistance device.

Block 608. Referring to block 608, in some implementations the measuredcolor of light is quantified at one or more wavelengths or wavelengthbands of light in the visible spectrum (e.g., detected as a wavelengthof 475 nanometers (nm), detected in a range of from 650 nm to 660 nm,etc.). For instance, in some implementations the measured color of lightis quantified in the visible spectrum (e.g., from approximately 390 nmto approximately 700 nm). In some implementations, the measured color oflight is also quantified in the infrared range (e.g., from approximately700 nm to approximately 500,000 nm).

Block 610. Referring to block 610, in some implementations the measuredcolor of light is quantified as a color temperature. This colortemperature refers to color characteristics of light. For instance, insome embodiments the color temperature is defined by a Kelvin ColorTemperature Scale, which defines light using a red to blue (e.g., warmto cool) scale. The Kelvin Color Temperature scale utilizes the lightthat is emitted through black body radiation of an object to assign anumerical value to each color. If color temperature is lower it will bemore red light and warmer. If color temperature is higher it will bemore blue light and cooler. Candle flames are 1900 K and castreddish/yellow light and are warm. Standard incandescent light is 2800 Kand warm. Daylight bulbs have color temperature 5000-6500 K and are coolwhite. Conventional digital screens tend to emit a color temperature ofabout 6500 k, which is equivalent to the color of light on a cloudy day.In some implementations, since the light conditions of an environmentvary throughout a day, the display of the device emit different types oflight to optimize the quality of light emitted by the display for theuser. For instance, warm white light is typically associated with acolor temperature range of from 2,000 K to 3,000 K, cool white light istypically associated with a color temperature range of from 3,100 K to4,500 K, and daylight is typically associated with a color temperaturerange of from 4,600 K to 6,500 K. In some implementations, the colortemperature scale is a correlated color temperature scale. Thiscorrelated color temperature scale is utilized for light sources thatemit discrete wavelengths and/or wavelength bands of light (e.g., do notdiscrete white light). For instance, in some implementations, the RGBsensor has a detection range of from about 2,856 K to about 6,500 K. Insome implementations, the RGB sensor has a detection range of from about2,850 K to about 6,550 K. In some implementations, the RGB sensor has adetection range of from about 2,600 K to about 6,850 K. Moreover, insome implementations, the RGB sensor has a detection range of from about2,000 K to about 7,000 K. Moreover, in some implementations, the RGBsensor has a detection range of from 1,800 K to 7,000 K.

Block 612. Referring to block 612, the one or more programs furtherinclude obtaining a measured brightness of light of the surroundingenvironment. Similar to the measured color of light, in someimplementations the measured brightness of light is determined fromlight of a surrounding environment detected by at least one sensor(e.g., RGB sensor 364 and/or camera 348 of FIG. 3 ) in the one or moresensors. A brightness refers to an overall darkness or lightness of adisplay of the device and/or the surrounding environment, which in someembodiments is measured in metric such as values of Lux. Further, insome implementations the measured brightness of light of the surroundingenvironment is detected with a fine granularity (e.g., a high degree ofprecision and/or accuracy). For instance, in some implementations themeasured brightness of light of a smart room environment 100 ranges fromand/or can be detected within a range of from approximately less thanone lux in dark environments (e.g., night) to approximately 10,000 lux,or more, in light environments (e.g., midday).

Block 614. Referring to block 614, in some implementations the measuredbrightness of light is quantified on a brightness scale. Similarly, insome implementations the measured brightness of light is quantified on aperceived brightness scale, which is related to the brightness scale.The brightness scale utilizes value that are measured from a luminancesensing device (e.g., ambient sensor 362 of FIG. 3 ), while theperceived brightness scale utilizes values of light as perceived by ahuman eye. In some implementations, the perceived brightness scale is afunction of the brightness scale. One such function includes convertingthe measured brightness of light into a perceived brightness by takingthe square root of the measured brightness, where the measuredbrightness is a percentage (e.g., one hundred percent represents amaximum plausible brightness of light, such as the sun or brightestsetting of a display and/or a lamp, and zero percent represents aminimum plausible brightness of light, such as complete darkness. Forinstance, in regard to a brightness of light that is emitted from adisplay of a display assistant device, in some implementations a user isallowed to manually adjust the brightness using an adjustment affordance(e.g., a dropdown menu, a sliding scale, etc.). Additional details andinformation regarding the perceived brightness of light and adjustingbrightness of light of the device are described in more detail in atleast blocks 630 through 634 of FIG. 6B.

Block 616. Referring to block 616, in some implementations the obtainingthe measured color of light and the obtaining the measured brightness oflight are concurrently conducted. For instance, in some implementationsthe ambient light sensor 362 and the RGB sensor 362 detect lightsimultaneously. Further, in some implementations the camera (e.g.,camera 348 of Figure), the ambient light sensor (e.g., ambient lightsensor 362 of Figure), and the RGB sensor (e.g., RGB sensor 362 of FIG.3 ) detect light simultaneously. Likewise, in some implementations theambient light sensor 362 and the RGB sensor 362 detect light in offsetperiods of time (e.g., a first sensor is active at a first time stampduring a period of time and a second sensor is active at a second timestamp distinct from the first stamp during the period of time.

Blocks 618 and 620. Referring to block 618, in some implementations theobtaining the measured color of light and the obtaining of the measuredbrightness of light are conducted on a recurring basis. For instance,referring to block 620, in some implementations the recurring basis isapproximately 5 second time intervals. In some implementations, therecurring basis is approximately 4 second time intervals. In someimplementations, the recurring basis is 2 second time intervals. In someimplementations, the recurring basis is approximately 1.5 second timeintervals. In some implementations, the recurring basis is approximately1.25 second time intervals. In some implementations, the recurring basisis approximately 1.0 second time intervals. In some implementations, therecurring basis is approximately 0.9 second time intervals. In someimplementations, the recurring basis is approximately 0.8 second timeintervals. In some implementations, the recurring basis is approximately0.7 second time intervals. In some implementations, the recurring basisis 0.6 second time intervals. In some implementations, the recurringbasis is approximately 0.5 second time intervals. In someimplementations, the recurring basis is approximately 0.4 second timeintervals. In some implementations, the recurring basis is approximately0.3 second time intervals. In some implementations, the recurring basisis approximately 0.2 second time intervals. In some implementations, therecurring basis is approximately 0.1 second time intervals. In someimplementations, the recurring basis is a time interval, where the timeinterval is between 500 milliseconds and ten minutes. In someimplementations, which are typically associated with shorter timeintervals of the recurring basis, the sensors of the device detect lighton a recurring basis that is repeated on at a predetermined frequency.For instance, in some implementations the sensors of the device areconfigured to detect light every five minutes, and do so using an abovedescribed recurring basis of 0.1 seconds.

Firing of the sensors in concurrent fashion allows for a more accurateand precise measurement to be detected, by yielding additional datapoints for each unit of time that the sensors are detecting. In someimplementations, the firing also increases a number of possiblemeasurements that are detected in a period of time, which can addadditional accuracy and/or precision to the quality of light emitted bythe device.

Blocks 622 and 624. Referring to block 622 of FIG. 6B, in someimplementations a color of light emitted from the display is adjusted bythe device. This color adjustment is in response to the obtaining of themeasured color and/or the measured brightness of light that is detectedby the one or more sensors of the device. Moreover, the adjustment ofcolor is from (i) an initial color of light emitted by the display ofthe device prior to the adjusting to (ii) a target color of light, wherethe target color of light matches the measured color of light. In otherwords, the device adjusts the overall color of the display to emit lightfrom a first color to a second color that is based off the measuredbrightness and/or color of light detected by the one or more sensors ofthe device. Referring to block 624, in some implementation the targetcolor of light is exactly equivalent to the measured color of light.However, the present disclosure is not limited thereto. For instance, insome implementations the target color of light is approximatelyequivalent to the measured color of light. For instance, in someimplementations the target color of light is offset by a predeterminedfactor, which will be described in more detail below.

In some implementations, the adjustment in the color of light emitted bya display is implemented using a first filter that is applied to eachrespective pixel of a pixelated image presently being displayed by thedisplay. In some implementations, the first filter acts to alter thenative red-green-blue (RGB) value of each pixel so that the pixel iscolor adjusted toward the target color of the light. In someimplementations, each pixel uses some form of additive color model otherthan the RGB color model. In some implementations, the first filter actsto alter the native color model value of each pixel so that the pixel iscolor adjusted toward the target color of the light. For instance, ifthe target color of light is 2500 K and the initial color of lightemitted by the display is 3000 K, the color values (e.g., RGB values) ofeach individual pixel of the pixelated image is altered so that,overall, the display of the image is normalized to 2500 K, rather thanthe original 2500 K. In some implementations, the initial color of lightemitted by a display is the average color of all the pixels presentlybeing displayed by the display. For instance, if the display isdisplaying an image that includes 1 million pixels, the initial color oflight emitted by the display is the average color across the 1 millionpixels. Accordingly, to adjust the display to the target color of light,the color of each of the 1 million pixels is proportionally adjusted byshifting the color of each pixel by an amount that will cause theaverage color across the 1 million pixels to be the target color oflight.

In some implementations, the brightness of light emitted by a display isimplemented using a brightness setting the applies equally to each pixelof the display.

In some implementations, the adjustment in the color of light emitted bya display is implemented using one or more display settings of thedisplay. For instance, in some implementations the one or more displaysetting parameters include an adjustment to a white point of thedisplay, which a set of chromaticity coordinates that serve to definethe color white in image capture, encoding, or reproduction. Using knownwhite point values of an illuminant, or estimating the white pointvalues, allows a picture to be adjusted from one illuminant to anotherilluminate through these known values. In some implementations, the oneor more display setting parameters include a color adjustment, such aschanging a hue of a color (e.g., a pure color), changing a tint of acolor (e.g., a hue with added white), changing a tone of a color (e.g.,a hue with added grey), and/or changing a shade of a color (e.g., a huewith added black). In some implementations, the one or more displaysetting parameters include a saturation of color, which is an intensityof a color. Moreover, in some implementations, the one or more displaysetting parameters include a contrast of pixels of the display, which isa difference between two different colors displayed on proximate pixels.These display setting parameters are dynamically (e.g., subtly) adjustedwith respect to the detected brightness and color in order to enhancethe display of media content on the device. Moreover, adjusting thedisplay setting parameters in some implementations modifies the colorsof a media content and/or tunes a media file based on the detectedqualities of light of the surrounding environment and/or the content ofthe media file itself. For instance, in some implementations mediacontent to be displayed on a device includes an image of food.Accordingly, adjusting the image of food to allow the food to pop in theimage display is desirable. As such, the display parameters may beadjusted to have the colors of the food more vibrant (e.g., brighter, ahigh contrast, more saturated, etc.). As another example, in someimplementations the media content to be displayed includes a castle orwinter landscape. As such, the display setting parameters are adjustedto provide more muted colors or reduced intensity an intensity of one ormore colors.

In some implementations, one or more display parameters can be adjustedbased on one or more of geo-location associated with the device, a timeassociated with the time (e.g., day or night context), a brightness oflight as detected from the surrounding environment, a color of light asdetected from the surrounding environment, a creation date and/or acreation location of media content, a desired mood associated with mediacontent (e.g., a picture of a rose is adjusted to be softer and warmer),content-specific factors, and other similar attributes of a mediafile/content. For instance, in some implementations based on thegeo-location of the user/device, it can be determined that a generalpreference in that particular location is for higher gloss and colorsaturation, and the display parameters of devices in that location areadjusted accordingly. In some implementations, display preferences for aregion or geo-location can be determined using crowdsourcing and/orother correlation methods. In some implementations, display settingparameters can be adjusted based on a day or night context to providefor dimming and/or compensating for light levels associated with thesetimes. In some implementations, display parameters can be adjusted, suchas adjusting white point, based on a real-time detected lighting of thesurrounding environment. In some implementations, content metadata canbe used to determine when and/or where content was captured, which isused to adjust display parameters, such as by adjusting to best displaythe season, weather, location, etc. at the time the content wascaptured. In some implementations, display parameters are adjusted basedon a desired mood, such as adjusting the white point to encourage a userto wake up in the morning or adjusting parameters to be more soothingwhile relaxing in the evening. In some implementations, displayparameters can be adjusted based on the sound environment of the device.For instance, if holiday music is playing the color tone can be adjustedas compared a color tone that would be used if rock music is playing inthe environment.

Moreover, in some implementations display setting parameters can also beadjusted based on the content itself. For instance, in displaying imagessuch as flowers, food, and/or the like, it may be desirable to adjustdisplay parameters to use more vibrant colors. As another example,display parameters may be adjusted to display winter scenes with moremuted colors whereas spring scenes may cause display parameters to beadjusted to provide more intensity. As another example, in displayingportraits, display parameters may be adjusted to display improved skintones and/or softer colors.

In some implementations, the factors used in identifying and adjustingdisplay parameters have a hierarchy or priority. For instance, in someimplementations detected light conditions of the surrounding environmentand/or content specific factors may be considered first in determininghow the display setting parameters are adjusted, with other factorshaving lower weight.

In some implementations, the surrounding environment of the displaydevice and/or data associated with the media content to be used inidentifying display parameters can be determined by the device itself,by a remote computing system (e.g., a cloud server, etc.), or by acombination of the device and the remote computing system. For instance,in some implementations a device provides the content and/or contextdata to a remote computing system and the remote computing system maydetermine one or more display parameters that should be adjusted on thedevice to enhance display of the content. For instance, the remotecomputing system can determine the subject of the content and/or thepreferences associated with the context of the environment, such as byusing machine-learned models, and identify one or more displayparameters that should be adjusted to enhance display of the content.Accordingly, the remote computing system provides data to the devicethat can be used to implement the display parameter adjustments.

Block 626. Referring to block 626, in some implementations a brightnessof light that is emitted from the display of the device is adjusted bythe device. This color adjustment is in response to the obtaining of themeasured color and/or the measured brightness of light detected by theone or more sensors of the device. Moreover, this brightness adjustmentis from an initial brightness of light that is emitted by the displayprior to the adjusting to a target brightness of light that matches themeasured brightness of light detected by the one or more sensors of thedevice. In other words, the device adjusts the brightness of the displayto emit light from a first brightness to a second brightness that isbased off the measured brightness and/or color of light detected by theone or more sensors of the device.

Block 628. Referring to block 628, in some implementations the adjustingthe brightness of light further includes referencing a lookup table thatis accessible to the device (e.g., lookup table module 337 of FIG. 3 ).The lookup table includes a first data field that includes one or morevalues of color of light (e.g., color data 338 of FIG. 3 ) and a seconddata field that includes one or more values of brightness of light(e.g., brightness data 339 of FIG. 3 ). However, the present disclosureis not limited thereto as, in some implementations, the lookup tableincludes additional data stores known to one skilled in the art of thepresent disclosure. Each value of brightness of light of the second datafield corresponds (e.g., relates) to at least one value of color oflight of the first data field. In some implementations, the lookup tableis useful as a correspondence between a color of light in the first datafield and a brightness of line in the second data field to determine thetarget color or the target brightness is necessary. Since an adjusted(e.g., outputted or emitted) brightness is a function of both a measuredcolor temperature and a measured ambient lux, the lookup table providesa reference to be used in determining the brightness to adjust to. Insome implementations, the function of the output brightness is linearwith respect to a measured with respect to values of a measured ambientbrightness (e.g., linear with respect to values of lux), and non-linearwith respect of values of a measured color temperature.

Block 630. Referring to block 630, in some implementations the targetbrightness of light is a brightness of light that is a user perceivedequivalence of the measured brightness of light. For instance, aspreviously described above, in some implementations the user perceivedequivalent of the measured brightness of light is a square root functionof the measured brightness of light. This function is based on a maximummeasured brightness of light as 100% (e.g., 10,000 lux) and a minimummeasured brightness of light at zero. For instance, in someimplementations if the user adjusts a brightness of the display, thebrightness is adjusted according to the perceived brightness function.For instance, if the user adjusts the brightness to a specific value ofthe maximum brightness of the display, the actual brightness is adjustedexponentially based off this value. For instance, if the user adjuststhe brightness to 90% of the maximum brightness of the display, theactual adjusted brightness of display is adjusted to 81%, since 0.9*0.9is 0.81. As another example, if the user adjusts the brightness to 45%of the maximum brightness of the display, the actual adjusted brightnessof display is adjusted to 20%, since 0.45*0.45 is approximately 0.20. Insome implementations, the user adjusts the brightness of the displaythrough a physical input of the device such as a toggle or slider,and/or a digital slider or dropdown menu, as well as a hand-free inputsuch as a vocal command.

As previously described, one aspect of the present disclosure provides adisplay of the digital assistant device with a high granularity suchthat a resolution of adjustable brightness is large. For instance, insome implementations the display supports a range of brightness of from0 candela per square meter (cd/m²) to 450 cd/m². In someimplementations, the display supports a range of brightness of from 0cd/m² to 425 cd/m². In some implementations, the display supports arange of brightness of from 0 cd/m² to 400 cd/m². In someimplementations, the display supports a range of brightness of from 0cd/m² to 375 cd/m². In some implementations, the display supports arange of brightness of from 0 cd/m² to 350 cd/m². Furthermore, in someimplementations the range of brightness is in steps of whole integers(e.g., 1 cd/m², 2 cd/m², . . . , 425 cd/m²). In some implementations,the range of brightness is divided into 8,192 equal steps. In someimplementations, the range of brightness is divided into 4,096 equalsteps. In some implementations, the range of brightness is divided into2,048 equal steps. In some implementations, the range of brightness isdivided into 1,024 equal steps.

In some implementations, the digital slider provides an instanttransition from an initial brightness to a selected brightness. In someimplementations, the digital slider provides a transition from aninitial brightness to a selected brightness that occurs over a period oftime (e.g., one second to five seconds). Additional details andinformation regarding the transitions of light will be described in moredetail below, with reference to at least block 636 of FIG. 6B.

In some implementations, the minimum adjusted brightness of light iscapped to a non-zero number. For instance, in some implementations, theminimum adjusted brightness of light is capped to 1.5% of the minimumbrightness of light emitted from the display. In some implementations,the minimum adjusted brightness of light is capped to 1% of the minimumbrightness of light emitted from the display. In some implementations,the minimum adjusted brightness of light is capped to 0.9% of theminimum brightness of light emitted from the display. In someimplementations, the minimum adjusted brightness of light is capped to0.8% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.7% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.6% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.5% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.4% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.3% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.2% of the minimum brightness of light emitted from the display. Insome implementations, the minimum adjusted brightness of light is cappedto 0.1% of the minimum brightness of light emitted from the display.

Blocks 632 and 624. Referring to block 632, in some implementations theuser perceived equivalence of the measured brightness of light is basedon an offset brightness relative to the measured brightness as detectedby the one or more sensors of the device. In some implementations, thisoffset of brightness accounts for inaccuracies in the detected qualitiesof light, and also provides the user with a means to add a permanentoffset (e.g., a user setting to generally make the display dimmer and/orbrighter). Referring to block 634, in some implementations the offsetbrightness is between 0.1% and 10% of the measured brightness. In someimplementations, the offset brightness is between 0.125% and 8% of themeasured brightness. In some implementations, the offset brightness isbetween 0.15% and 6% of the measured brightness. In someimplementations, the offset brightness is between 0.25% and 4% of themeasured brightness. These offset values are capped at 100% of themaximum brightness of light emitted by the display. The offset valuesare floored at 0.05% of the maximum brightness of light emitted by thedisplay. In some implementations, the offset values are floored at 0.1%of the maximum brightness of light emitted by the display. In someimplementations, the offset values are floored at 0.2% of the maximumbrightness of light emitted by the display. In some implementations, theoffset values are floored at 0.2% of the maximum brightness of lightemitted by the display. In some implementations, the offset values arefloored at 0.5% of the maximum brightness of light emitted by thedisplay. In some implementations, the offset floor values are dependentupon a respective mode and/or state of the device. These floor capsprevent the display from becoming too dark that a user cannot seeinformation on the display. Moreover, in some implementations the capsprevent the offset from affecting one or more modes and/or states of thedevice (e.g., a low-light clock).

In some implementations, an offset that is applied to a brightness ofthe display if the display is not engaged is determined according to abrightness of the display in an engaged state. For instance, in someimplementations the brightness of the display in an engaged state iscategorized into one or more ranges of brightness (e.g., a first rangeof 100% to 25%, a second range of 24% to 20%, etc.). Table 1, shownbelow, provides an implementation of such ranges and associatedunengaged brightness of the device.

Engaged Brightness Unengaged Brightness of the Display of the Display25% to 100% 25% to 100% 24.99% to 20% 24.99% to 16% 19.99% to 13% 15.99%to 9% 12.99% to 5% 8.99% to 4% 4.99% to 2.5% 3.99% to 2% 2.49% to 1.25%1.99% to 1% 1.24% to 0% .99% to 0%

One skilled in the art might appreciate other ranges of engagedbrightness and associated ranges of unengaged brightness that might beused in accordance with the present disclosure.

In some implementations, the user adjusts the applied offset using adigital slider or other similar mechanic such as a drop down menu.Further, in some implementations the digital slider has a weighted zerovalue so as to attach (e.g., snap to) a user selection towards the zero,default offset. In some implementations, this zero default offset is amode that configures the display of the device to replicate anon-digital picture frame. In some implementations, the default offsetis slightly greater than zero so as to allow media content that is beingdisplayed to appear more vividly. In some implementations, the offsetvalues applied to each device are unique to each device. For instance,in some implementations a device that is installed in a dark basementhas a dimmer brightness setting compared to a device that is installedin a bright sun room. Moreover, in some implementations the offsetvalues applied to each device are specific to a respective user of thedevice.

Block 636. Referring to block 636, in some implementations the adjustingthe color of light emitted from the display and the adjusting thebrightness of light emitted from the display are implemented as atransition. This transition between an initial color of light emittedfrom the display prior to the adjusting and the target color. Moreover,the transition is also between an initial brightness of light emittedfrom the display prior to the adjusting and the target brightness. Insome implementations, the transition of the color and the transition ofthe brightness occur simultaneous. However, the present disclosure innot limited thereto. For instance, in some implementations thetransition of color and the transition of brightness occur independentof each other, such as in a calibration mode of the display of thedevice. In some implementations, the transitions occur over apredetermined period of time. For instance, in some implementation thetransitions occur over of five second period of time. In someimplementation, the transitions occur over a four second period of time.In some implementation, the transitions occur over a three second periodof time. In some implementation, the transitions occur over a two secondperiod of time. In some implementation, the transitions occur over asecond period of time. To this point, in some implementations thesetransitions from the initial brightness and/or the initial color to theadjusted brightness and/or the adjusted color are based on a function oftime. For instance, in some implementations the function is linear(e.g., a transition from 0.15% brightness to 0.25% brightness over afive second period occurs in 0.02% intervals). In some implementations,the function is an exponential function such as a cubic function. Acubic function of transition can yield a more natural transition to theeye of the user. Moreover, in some implementation the transitions occurinstantaneously.

In some implementations, if the display is transitioning between a firstset of brightnesses and/or colors and a new set of brightness and/orcolor is determined (e.g., provided by the user or determined by thedevice), the transition of the first set is interrupted and a newtransition is initiated to arrive at the newly determined brightnessand/or color. In other words, when a transition is interrupted, a newtransition in initiated upon receipt of the interruption.

Block 638. Referring to block 638 of FIG. 6C, in some implementationsthe one or more programs include triggering a device state according toone or more determinations and/or condition. In some implementations, adevice state is triggered in accordance with a determination that themeasured brightness of light satisfies a first threshold brightnessvalue. For instance, in some implementations a first device state (e.g.,a dim light state) is triggered in accordance with a determination thatthe measured brightness of light is at or below a threshold value of 1%of the maximum brightness of the display. In some implementations, thefirst device state is triggered in accordance with a determination thatthe measured brightness of light is at or below a threshold value of0.9% of the maximum brightness of the display. In some implementations,the first device state is triggered in accordance with a determinationthat the measured brightness of light is at or below a threshold valueof 0.8% of the maximum brightness of the display. In someimplementations, the first device state is triggered in accordance witha determination that the measured brightness of light is at or below athreshold value of 0.7% of the maximum brightness of the display. Insome implementations, the first device state is triggered in accordancewith a determination that the measured brightness of light is at orbelow a threshold value of 0.6% of the maximum brightness of thedisplay. In some implementations, the first device state is triggered inaccordance with a determination that the measured brightness of light isat or below a threshold value of 0.5% of the maximum brightness of thedisplay.

Furthermore, in some implementations, a second device state (e.g., adark light state) is triggered in accordance with a determination thatthe measured brightness of light is at or below a threshold value of0.2% of the maximum brightness of the display. In some implementations,the second device state is triggered in accordance with a determinationthat the measured brightness of light is at or below a threshold valueof 0.175% of the maximum brightness of the display. In someimplementations, the second device state is triggered in accordance witha determination that the measured brightness of light is at or below athreshold value of 0.15% of the maximum brightness of the display. Insome implementations, the second device state is triggered in accordancewith a determination that the measured brightness of light is at orbelow a threshold value of 0.15% of the maximum brightness of thedisplay. In some implementations, the second device state is triggeredin accordance with a determination that the measured brightness of lightis at or below a threshold value of 0.125% of the maximum brightness ofthe display. In some implementations, the second device state istriggered in accordance with a determination that the measuredbrightness of light is at or below a threshold value of 0.1% of themaximum brightness of the display. In some implementations, the seconddevice state is triggered in accordance with a determination that themeasured brightness of light is at or below a threshold value of 0.075%of the maximum brightness of the display. In some implementations, thesecond device state is triggered in accordance with a determination thatthe measured brightness of light is at or below a threshold value of0.05% of the maximum brightness of the display.

Block 640. Referring to block 640, in some implementations a thirddevice state disables the display. For instance, in some implementationsthe third device state disables the device in accordance with adetermination that the device satisfies a threshold value of time sincea previous engagement by a user (e.g., the device will timeout after aperiod of time in which the user does not engage with the device). Insome implementations, the threshold value of time is five seconds. Insome implementations, the threshold value of time is ten seconds. Insome implementations, the threshold value of time is five minutes. Insome implementations, the threshold value of time is ten minutes. Insome implementations, the threshold value of time is thirty minutes. Insome implementations, the threshold value of time is an hour. In someimplementations the displaying of the display includes completelyturning off the display or setting a brightness of the display to zero.In some implementations, the device remains in the third state until auser engages with the device or a threshold value of brightness issatisfied. This third device state allows for the device to minimize itsexcess energy consumption due to being left on when a user is notengaging with the device and/or when the user is passively engaging withthe device, such as listening to music, playing a video, and/or runninga countdown timer.

Block 642. Referring to block 642, in some implementations a fourthdevice state displays predetermined information on the display. Forinstance, in some implementations the fourth mode of the device displaysa clock and/or an alarm indicator. In some implementations, the clock isdisplayed in a variety of fonts, such as block letters, a variety ofcolors, such as white or light blue, and/or a variety of types, such as24 hour or 12 hour types. In some implementations, the alarm indicatoris a digital indicator such as text and/or an icon, and/or may be aphysical indicator such as an LED indicator. In some implementations,the alarm indicator is active in accordance with a determination that analarm is scheduled to trigger within a predetermined period of time(e.g., a day).

In some implementations, one or more states of the device do not utilizethe color adjustment aspects of the present disclosure. For instance, insome implementations in a dim state, a dark state, and/or an ambientstate the color of light emitted by the device is capped at apredetermined value and/or limited to a predetermined range. In someimplementations, this value and/or range is determined by the user ordetermined a setting of the device. In some implementations, thepredetermined value is approximately 6,400 K. In some implementations,the predetermined value is approximately 6,450 K. In someimplementations, the predetermined value is approximately 6,500 K. Insome implementations, the predetermined value is approximately 6,550 K.of the device. In some implementations, the predetermined value isapproximately 6,600 K. In some implementations, the predetermined valueis approximately 6,650 K. In some implementations, the predeterminedvalue is approximately 6,700 K.

Block 644. Referring to block 644, in some implementations the one ormore programs include removing, in accordance with a determination thatthe measured brightness satisfies a second threshold brightness value,the device from the device state. The second threshold brightness valuebeing greater than or equal to the first threshold brightness value. Forinstance, the first device state is cancelled (e.g., stopped) inaccordance with a determination that the measured brightness of light isat or above a threshold value of 1.5% of the maximum brightness of thedisplay. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of 1.4% of the maximum brightness of thedisplay. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of 1.3% of the maximum brightness of thedisplay. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of 1.2% of the maximum brightness of thedisplay. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of 1.1% of the maximum brightness of thedisplay. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of 1.0% of the maximum brightness of thedisplay. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of a 200% of the triggering thresholdvalue of the first state. In some implementations, the first devicestate is cancelled in accordance with a determination that the measuredbrightness of light is at or above a threshold value of a 175% of thetriggering threshold value of the first state. In some implementations,the first device state is cancelled in accordance with a determinationthat the measured brightness of light is at or above a threshold valueof a 160% of the triggering threshold value of the first state. In someimplementations, the first device state is cancelled in accordance witha determination that the measured brightness of light is at or above athreshold value of a 150% of the triggering threshold value of the firststate. In some implementations, the first device state is cancelled inaccordance with a determination that the measured brightness of light isat or above a threshold value of a 140% of the triggering thresholdvalue of the first state. In some implementations, the first devicestate is cancelled in accordance with a determination that the measuredbrightness of light is at or above a threshold value of a 125% of thetriggering threshold value of the first state. In some implementations,the first device state is cancelled in accordance with a determinationthat the measured brightness of light is at or above the triggeringthreshold value of the first state.

Furthermore, the second device state is cancelled (e.g., stopped) inaccordance with a determination that the measured brightness of light isat or above a threshold value of 0.3% of the maximum brightness of thedisplay. In some implementations, the second device state is cancelledin accordance with a determination that the measured brightness of lightis at or above a threshold value of 0.25% of the maximum brightness ofthe display. In some implementations, the second device state iscancelled in accordance with a determination that the measuredbrightness of light is at or above a threshold value of 0.2% of themaximum brightness of the display. In some implementations, the seconddevice state is cancelled in accordance with a determination that themeasured brightness of light is at or above a threshold value of 0.15%of the maximum brightness of the display. In some implementations, thesecond device state is cancelled in accordance with a determination thatthe measured brightness of light is at or above a threshold value of0.1% of the maximum brightness of the display. In some implementations,the second device state is cancelled in accordance with a determinationthat the measured brightness of light is at or above a threshold valueof a 200% of the triggering threshold value of the second state. In someimplementations, the second device state is cancelled in accordance witha determination that the measured brightness of light is at or above athreshold value of a 175% of the triggering threshold value of thesecond state. In some implementations, the second device state iscancelled in accordance with a determination that the measuredbrightness of light is at or above a threshold value of a 160% of thetriggering threshold value of the second state. In some implementations,the second device state is cancelled in accordance with a determinationthat the measured brightness of light is at or above a threshold valueof a 150% of the triggering threshold value of the second state. In someimplementations, the second device state is cancelled in accordance witha determination that the measured brightness of light is at or above athreshold value of a 140% of the triggering threshold value of thesecond state. In some implementations, the second device state iscancelled in accordance with a determination that the measuredbrightness of light is at or above a threshold value of a 125% of thetriggering threshold value of the second state. In some implementations,the second device state is cancelled in accordance with a determinationthat the measured brightness of light is at or above the triggeringthreshold value of the second state.

Having the cancellation threshold value be greater than the triggeringthreshold value for the one or more modes of the device allows therespective mode of the device to remain active through inadvertent orsudden spikes in detected light, such as a TV, which is proximate to thedisplay assistant device, suddenly displaying a bright white sceneimmediately after displaying a dark grey scene.

Blocks 646 and 648. Referring to block 646, in some implementations thedevice further includes a microphone (e.g., microphone 342 of FIG. 3 )and one or more speakers (e.g., speaker 350 of FIG. 3 ). Accordingly,the one or more programs further include outputting a pulse of soundthrough the one or more speakers of the device. Responsive to theoutputting, the pulse of sound is received through the microphone afterbeing reflected off one or more objects (e.g., walls, users of thedevice, etc.) in the surrounding environment. In some embodiments, theone or more programs also include determining, responsive to thereceiving of the pulse of sound, if one or more users of the device islocated in the surrounding environment. This determination is based onan amount of time that elapses between the outputting and the receivingof the sound after reflecting off one or more objects. Referring toblock 648, in some implementations the pulse of sound is inaudible(e.g., an ultrasound) to the user of the device. This allows for thedevice to detect the presence of a user without additional costlysensors and equipment. Additional details and information can be foundin international patent application number PCT/US2018/04878, entitled“Systems and Methods for Ultrasonic Sensing of Persons in a HomeEnvironment,” which is hereby incorporated by reference in its entirety.This reference provides on skilled in the art with further details andinformation that is necessary, in some embodiments, to detect one ormore objects in the surrounding environment of the display assistantdevice of the present disclosure.

Block 650. Referring to block 650, in some implementations a user of thedevice overrides the adjusting to cause the display to emit a brightnessof light specified by the user. In some implementations, this overrideis as described above with respect to the offset provided by the user.In some implementations, this override is as described above withrespect to the manual adjustment of the brightness of the displayprovided by the user. In some implementations, a user can further modifythe display setting parameters based on user preferences. In otherwords, in some implementations the brightness and/or color of lightemitted by the display are automatically determined according to thesystems and methods of the present disclosure or are manually adjustedby the user.

Block 652. Referring to block 652, in some implementations the one ormore programs include determining when a user is engaged with thedevice. In accordance with a determination that the device is currentlyengaged, a state (e.g., a fifth state) of the device is activated (e.g.,an engaged state of the device). In some implementations, the fifthstate of the device is associated with a particular brightness of lightemitted from the display of the device, which exceeds the targetbrightness. In accordance with a determination that the device iscurrently unengaged, the device is triggered to return to emitting lightat the target brightness. In other words, this state allows the displayof the digital assistant device to brighten if a user is engaged withthe device, and dim if a user is no longer engaged with the device.

Block 654. Referring to block 654, in some implementations theengagement with the device by the user includes a vocal interaction,which is received through a microphone (e.g., microphone 342 of FIG. 3 )of the device. In some implementations, the engagement with the deviceby the user includes a touch interaction, which is received through thedisplay (e.g., display 352 of FIG. 3 ) of the device. In someimplementations, the display is a touch sensitive display. Moreover, insome implementations the engagement with the digital assistant device bythe user includes an auxiliary interaction. In some implementations adetection of a presence of the user by one of more sensors of thedigital assistant device constitutes an auxiliary interaction. Forinstance, in some implementations the digital assistant device is in adark room. Upon detection of the presence of a user of the device, thedigital assistant device will brighten the display to illuminate aportion of the surrounding environment, allowing the user to safelynavigate the environment in the dark without human intervention. In someimplementations, the detection is conducted by the above describedspeaker and microphone systems and methods, with reference to at leastblocks 646 and 648 of FIG. 6C.

Furthermore, in some implementations in accordance with a determinationthat the device is currently engaged, a sixth state of the device isactivated. This sixth state is distinct from the fifth state of thedevice in that the associated brightness of the sixth state is less thana target brightness. For instance, in some implementations a userinterferes with a source of light and the digital assistant device(e.g., a user stands interposing between the light source and thedevice). This interference by the user interrupts the light that isdetected by one or more sensors of the device, producing a darker thannormal display. Furthermore, in some implementations the detection ofthe user is conducted by the above described speaker and microphonesystems and methods, with reference to at least blocks 646 and 648 ofFIG. 6C.

In some implementations, the auxiliary interaction is either receivedfrom a remote computer system and/or provided through one or moreprograms of the device. For instance, in some implementations theauxiliary intervention is a high priority notification, such as amessage marked with high important or an incoming phone call provided bythe remote computer system. Moreover, in some implementations the highpriority notification is an alarm, a timer, and/or an alert that isstored in the one or more programs of the digital assistant device. Ifthe auxiliary intervention is triggered by one of these high prioritynotifications, the device will remain in the present state until thenotification is longer active in some implementations. Similarly, insome implementations if the notification is longer active the devicereverts to its previous state (e.g., a state before receiving thenotification).

Block 656. Referring to block 656 of FIG. 6D, in some implementationsthe determination that the device is currently unengaged occurs inaccordance with a determination that a previously received engagementwith the device satisfies a threshold period of time. For instance, insome implementations in accordance with a determination that a user hasnot engaged the device for a period of five minutes, the device isdetermined to be unengaged. In some implementations, in accordance witha determination that a user has not engaged the device for a period offifteen minutes, the device is determined to be unengaged. In someimplementations, in accordance with a determination that a user has notengaged the device for a period of thirty minutes, the device isdetermined to be unengaged. In some implementations, in accordance witha determination that a user has not engaged the device for a period ofan hour, the device is determined to be unengaged.

Block 658. Referring to block 658, in some implementations the devicefurther incudes a camera (e.g., camera 348 of FIG. 3 ). Accordingly, theone or more programs include determining the color of light of thesurrounding environment determining from measured light captured by thecamera. The one or more programs also includes determining thebrightness of light of the surrounding environment from measured lightcaptured by the camera. This determining is in accordance with adetermination that at least one of the measured color and the measuredbrightness of light satisfies a threshold value of confidence. Forinstance, in some implementations the threshold value is satisfied if anaccuracy and/or precision of a detected value of light of thesurrounding environment is an outlier compared to previous detections.Accordingly, using the determined color of light of the surroundingenvironment as detected by the one or more sensors and as captured bythe camera, the target color of light of the display is determined.Furthermore, using the determined brightness of light of the surroundingenvironment as detected by the one or more sensors and as captured bythe camera, the target brightness of light is determined. The determinedtarget color and target brightness of the display have a higher degreeof confidence since the camera augments and/or verifies the values thatare detected by the other sensors of the device.

Block 660. Referring to block 660, in some implementation the one ormore programs include displaying a media file on the display. A type ofthe media file is then determined by the device (e.g., the media file isa video, the media file is a picture, etc.). Assuring, in accordancewith a determination that the type of media file is a digital image, thebrightness of light emitted from the display of the device satisfies afirst threshold brightness (e.g., 1% of maximum brightness of thedisplay, 2% of maximum brightness of the display, 20% of maximumbrightness of the display, . . . , 10%, etc.). Assuring, in accordancewith a determination that the type of media file is a digital video, thebrightness of light emitted from the display of the device satisfies asecond threshold brightness (e.g., 20% of maximum brightness of thedisplay, 25% of maximum brightness of the display, 20% of maximumbrightness of the display, etc.). In some implementations, in accordancewith a determination that a user has not engaged the device for a periodof time and the device is playing a video, the device is determined tobe engaged. These determinations in regards to videos playing on thedevice allow for the device to remain illuminated if the device isplaying a video with an extended duration (e.g., longer than a time-outsetting of the device), and to have an illumination to satisfactorilyrender the video, which typically suffer from poor display quality inlow light scenes displayed on a low brightness display.

Furthermore, in some implementations if a media file is being displayedon the device, a content of the media file is determined. The content ofthe media files includes a brightness of the media file, which is usedto determine an adjusted brightness of the display (e.g., a media filethat depicts a skyline at night is associated with a darker brightnessas compared to a media file that depicts a sunny beach).

In some implementations, the digital assistant device communicates withthe smart home environment in order to determine characteristics oflight that are emitted from smart lights of the environment (e.g., smartlights 124 of FIG. 1 ). For instance, if a smart light is determined tooutput a specific color temperature or wavelength with a particularbrightness, this information can be communicated to the device. Thisallows for the device to utilize the information provided by the smartlight instead of having to detect the light emitted by the smart light.

Moreover, in some implementations the digital assistant device and/orthe smart home environment stores a log of brightness and/or colors oflight emitted by the device throughout a period of time. This allows thedevice to reference the log and determine one or more patterns in thebrightness and/or colors of light emitted by the device, in order toprovide a display that matches preferences of each user and/or homesetting. Additional details and information can be found in U.S. patentapplication Ser. No. 14/581,994, entitled “Apparatus and Method forProgramming and Controlling Devices in the Home with Sensor Data,Learning, and Repetition,” which is hereby incorporated by reference inits entirety. This reference provides on skilled in the art with furtherdetails and information which is necessary to determine patterns ofsmart devices in the surrounding environment of the display assistantdevice in accordance with some embodiments of the present disclosure.

In some implementations, the device includes a seventh state that isprovided when the device is turned on. The seventh state is configuredto set the brightness of the display to 100% maximum brightness of thedisplay, which allows for the user to visualize the power and brightnessof the display and/or for calibration purposes.

In some implementations, the one or more programs include determining alocation of the device. In some implementations, the location of thedevice is provided by a user of the device (e.g., through a setting ofthe device), or be provided through a remote computer system (e.g.,detected through communications with the remote computer system such asan internet protocol address). Moreover, in some implementations thelocation of the device includes an associated time and/or date (e.g., alocation determined as San Francisco, California will determine a timeas pacific standard or daylight time), or the time and/or date may bedetermined by the device independent of the determined location. In someimplementations, determining the location of the device providesinformation related to an anticipated sunrise and an anticipated sunsetof the location. In some implementations, the above describedinformation (e.g., location, time, sunrise and sunset) is used todetermine a brightness of the device. This allows for the device toadjust the brightness of the display in accordance with a location ofthe device and expected outdoor brightness of the location.

Furthermore, in some implementations the device includes an eighth modethat places the device in a privacy configuration. In someimplementations, the privacy configuration includes disabling the cameraand/or the microphone of the device in order to provide a user of thedevice with an assurance that their presence will not be accidentallycaptured by the device. Moreover, in some implementations the eighthmode includes an indicator (e.g., an LED indicator) installed in thedevice which signals that the mode is either active or inactive. To thispoint, in some implementations an emitted brightness of light of thedisplay is signaled by an indicator r (e.g., an LED indicator) installedin the device which signals that the level of brightness of the device.For instance, in some implementations the LED is off at a 0% brightnessof the display and the LED is at 100% power at a 100% brightness of thedisplay. Furthermore, in some implementations each LED indicatordescribed by the present disclosure has a respective brightness that isdetermined by an emitted brightness of the display (e.g., if the displayis at 50% brightness, one or more of the LED indicators will be at 50%brightness as well).

Referring to FIG. 7A, in some implementations a display assistant device(e.g., device 700-1) includes a settings menu to allow a user of thedevice to manipulate various configurations of the device. The settingsmenu includes general settings 702, such as modifying a device name,modifying an associated room of the device, and modifying a networkconnection of the device. The setting menu also includes device settings704, such as modifying a speaker setting of the digital assistantdevice, modifying a media output device setting of the digital assistantdevice, modifying various alarms and timers of the device (e.g., highpriority notifications), modifying a display setting of the device,and/or modifying a mode or state of the device. In FIG. 7A, dot 706indicates a use input (e.g., a touch input), which leads to the userinterface of FIG. 7B.

Referring to FIG. 7A, in some implementations the display settings ofthe device are further modified according to the systems and methodsdescribed by the present disclosure. For instance, in someimplementations the display settings include general settings 710 suchas determining a threshold of a low light mode (e.g., a dim mode or adark mode), and what type of information do be displayed by the lowlight mode, as well as a timeout setting of the display. Moreover, insome implementations the display settings include light equalizersettings 712 (e.g., color and brightness settings of the display), suchas applying a user defined offset of brightness and/or a coloradjustment setting of the modes of the device. One of skill in the artof the present disclosure will appreciate that other device settings anddisplay settings are plausible that are not illustrated in FIG. 7 .

Example (i)—Digital Assistant Device Control

TABLE 2 Actions and Results of a Control of the Digital Assistant DeviceAction Result Device Setup 100% brightness until device goes intoAmbient for a first time, in which Auto-brightness is turned on. ToggleAuto-brightness ON Device matches a surrounding environment. Brightnessindicator greyed out 2 second transition Toggle Auto-brightness OFFDevice at fixed brightness. Brightness indicator indicates currentbrightness. .5 second transition. Will store previous manual brightnesssetting. Change brightness manually Switch to manual brightness mode.while Auto-brightness ON Adjust brightness accordingly Brightnessadjusts according to perceived brightness Change brightness manuallyAdjust brightness accordingly while Auto-brightness OFF Brightnessadjusts according to perceived brightness Set brightness to MAXIMUMBrightness should be set to 100% Set brightness to MINIMUM Brightnessshould be set to 0.6% Power-cycle device with Auto- Will restart withAuto-brightness OFF and last brightness OFF brightness settingPower-cycle device with Auto- Will restart with auto-brightness ONbrightness ON Voice commands: Set the Will set the perceived brightnessto a particular level, brightness to X and a corresponding number ofvalue on a slider should be highlighted on device. Quick Settings traywill appear and show updated brightness if an in-between percentage(e.g. set the brightness to 75) should

Example (ii)—Digital Assistant Device Behavior

TABLE 3 Actions and Results of a Behavior of the Digital AssistantDevice Action Result Change the lighting in the Brightness and colortemperature will change. environment Within 700 ms transition willstart. Will end after 4 seconds. Reduce lighting to go into low Willrespect different entrance and exit thresholds. light clock mode. Thenincrease Independent of whether or not Auto-brightness is lighting toleave it. ON. If Auto-brightness is OFF, it always uses the DARKthreshold. Make a voice query while in Device will switch to engagedbrightness during Ambient mode or low light clock query, then back tounengaged when done if still on Ambient mode or low light clock mode.Wiggle the screen while on low Device will switch to engaged brightnessfor 10 light clock mode seconds. Color temp will not change. Wiggle orswipe the screen while Device should switch to engaged brightness for 10on Ambient mode seconds. Color temp will not change. Tap the screenwhile on low light Device will go to Ambient mode at engaged clockbrightness. Tap the screen while on Ambient Will go to Home Screen andan engaged mode brightness. Pull up/down a tray while on low Will go toan engaged brightness while the tray is light clock showing Pull up/downa tray while on Will go to an engaged brightness while the tray isAmbient mode showing Swipe back from Ambient to low Device will stay atunengaged brightness. light clock Play a video Brightness will befloored at 25% as long as video is full-screen. Play a video then goback to Device will go back to engaged brightness. Home Play a videothen go back to Device will go back to unengaged brightness. Ambient orlow light clock mode Play music Device will go to engaged brightness.Play music and then go back to Device will stay at engaged brightness.Home Play music and then go back to Device will go back to unengagedbrightness. Ambient or low light clock mode Show an error screen Devicewill stay on whatever brightness it was previously at. Show anotification (alarm, timer, Device will switch to engaged brightness forthe incoming call) duration of the interruption. Swipe back into lowlight clock Device will turn on low light clock mode and modeAuto-brightness Will not leave this state until the user engages withthe device or there's a notification.

Example (iii)—Digital Assistant Home State

TABLE 4 Actions and Results of a Home State of the Digital AssistantDevice Action Result Set Low light activation to Screen shifts tolow-light mode when environment DIM gets to 0.8% brightness Set Lowlight activation to Screen shifts to low-light mode when environmentDARK gets to 0.1% brightness Set During Low light to When the devicegoes into low-light mode, it shows SHOW CLOCK the low-light clock. SetDuring Low light to TURN When the device goes into low-light mode, itturns OFF SCREEN the screen off. Toggle Screen Timeout ON After 5minutes of inactivity, the screen turns off. Toggle Screen Timeout OFFNothing happens after 5 minutes of inactivity. Adjust the BrightnessOffset The user is able to see the screen getting brighter. BrighterThis will not be noticeable in very bright conditions, and will be verynoticeable in low-light conditions. The max offset is +6%. This isapplied at the 2% brightness level. Adjust the Brightness Offset Theuser is able to see the screen getting dimmer. Dimmer This will not benoticeable in very bright conditions, and yy ill be more noticeable inlow-light conditions. The max offset is −1%. This is applied at the 2%brightness level. Check the default Brightness Set to 2% Offset Set theBrightness Offset to the The device is indistinguishable from an actualmiddle physical photo behind glass. Set Color Matching to The devicechanges to ambient mode and all other ALWAYS modes to match colortemperature of the environment. Set Color Matching to The device changesonly ambient to match color AMBIENT ONLY temperature of the environment.Other modes are always white. Set Color Matching to NEVER The devicenever matches the color temperature of the environment.

The terminology used in the description of the various describedimplementations herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used in thedescription of the various described implementations and the appendedclaims, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof Additionally, it will be understood that, althoughthe terms “first,” “second,” etc. may be used herein to describe variouselements, these elements should not be limited by these terms. Theseterms are only used to distinguish one element from another.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

It is to be appreciated that “smart home environments” may refer tosmart environments for homes such as a single-family house, but thescope of the present teachings is not so limited. The present teachingsare also applicable, without limitation, to duplexes, townhomes,multi-unit apartment buildings, hotels, retail stores, office buildings,industrial buildings, and more generally any living space or work space.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the claims to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings. The embodimentswere chosen and described in order to best explain principles ofoperation and practical applications, to thereby enable others skilledin the art.

Although various drawings illustrate a number of logical stages in aparticular order, stages that are not order dependent may be reorderedand other stages may be combined or broken out. While some reordering orother groupings are specifically mentioned, others will be obvious tothose of ordinary skill in the art, so the ordering and groupingspresented herein are not an exhaustive list of alternatives. Moreover,it should be recognized that the stages can be implemented in hardware,firmware, software or any combination thereof.

The above description, for purpose of explanation, has been describedwith reference to specific implementations. However, the illustrativediscussions above are not intended to be exhaustive or to limit thescope of the claims to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theimplementations were chosen in order to best explain the principlesunderlying the claims and their practical applications, to therebyenable others skilled in the art to best use the implementations withvarious modifications as are suited to the particular uses contemplated.

What is claimed is:
 1. A method of adjusting light emitted from adisplay of a computerized device, the method comprising: based on lightdetected by one or more sensors, measuring, by the computerized device,a color of the light; based on the light detected by the one or moresensors, measuring, by the computerized device, a brightness of thelight; determining, by the computerized device, a plurality oftransitions to perform, wherein the plurality of transitions comprises:a first transition that adjusts a color of light emitted from thedisplay from an initial color of light emitted by the display prior tothe adjusting to a target color of the light that matches the measuredcolor of the light; and a second transition that adjusts a brightness oflight emitted from the display from an initial brightness of lightemitted by the display prior to the adjusting to a target brightness ofthe light that matches the measured brightness of the light; andcausing, by the computerized device, the plurality of transitions tooccur.
 2. The method of claim 1, further comprising: after causing theplurality of transitions to occur, determining, by the computerizeddevice, that a user is engaged with the computerized device; and basedon determining that the user is engaged with the computerized device,adjusting the brightness of light emitted from the display from thetarget brightness to a brightness level that exceeds the targetbrightness.
 3. The method of claim 2, further comprising: afteradjusting the brightness of light emitted from the display to thebrightness that exceeds the target brightness, determining, by thecomputerized device, that the user is no longer engaged with thecomputerized device; and based on determining that the user is no longerengaged with the computerized device, adjusting the brightness of lightemitted from the display back to the target brightness from thebrightness level.
 4. The method of claim 3, wherein the user beingengaged with the computerized device comprises a vocal interaction. 5.The method of claim 3, wherein the user being engaged with thecomputerized device comprises a touch interaction performed by the useron the display.
 6. The method of claim 1, wherein the first transitionand the second transition are performed simultaneously.
 7. The method ofclaim 1, further comprising: after causing the plurality of transitionsto occur, determining that a second brightness of the light measuredusing the one or more sensors satisfies a first threshold brightnessvalue; and in response to determining that the second brightness of thelight measured using the one or more sensors satisfies the firstthreshold brightness value, presenting predetermined information on thedisplay.
 8. The method of claim 7, wherein the predetermined informationcomprises a clock.
 9. The method of claim 8, wherein the predeterminedinformation further comprises an alarm indicator.
 10. The method ofclaim 1, wherein the adjusting the brightness of the light furthercomprises: referencing a lookup table that is accessible to thecomputerized device, the lookup table comprising: a first data fieldthat includes one or more values of color of light, and a second datafield that includes one or more values of brightness of light, whereineach value of brightness of light of the second data field correspondsto at least one value of color of light of the first data field; andusing a correspondence between a color of light in the first data fieldand a brightness of light in the second data field to determine thetarget color or the target brightness.
 11. A computerized devicecomprising: one or more processors; a touch screen display; one or moresensors that detect light in an ambient environment of the computerizeddevice; and a memory that stores instructions that, when executed by theone or more processors, cause the one or more processors to: determine acolor of the light based on light detected by one or more sensors;determine a brightness of the light based on the light detected by theone or more sensors; determine a plurality of transitions to perform,wherein the plurality of transitions comprises: a first transition thatadjusts a color of light emitted from the touch screen display from aninitial color of light emitted by the touch screen display prior to theadjusting to a target color of the light that matches the measured colorof the light; and a second transition that adjusts a brightness of lightemitted from the touch screen display from an initial brightness oflight emitted by the touch screen display prior to the adjusting to atarget brightness of the light that matches the measured brightness ofthe light; and cause the plurality of transitions to occur on the touchscreen display.
 12. The computerized device of claim 11, wherein theinstructions further cause the one or more processors to: determine thata user is engaged with the computerized device after causing theplurality of transitions to occur; and adjust the brightness of lightemitted from the touch screen display from the target brightness to abrightness level that exceeds the target brightness based on determiningthat the user is engaged with the computerized device.
 13. Thecomputerized device of claim 12, wherein the instructions further causethe one or more processors to: after adjusting the brightness of lightemitted from the touch screen display to the brightness that exceeds thetarget brightness, determine that the user is no longer engaged with thecomputerized device; and adjust the brightness of light emitted from thetouch screen display back to the target brightness from the brightnesslevel based on determining that the user is no longer engaged with thecomputerized device.
 14. The computerized device of claim 13, whereinthe user being engaged with the computerized device comprises a vocalinteraction.
 15. The computerized device of claim 13, wherein the userbeing engaged with the computerized device comprises a touch interactionperformed by the user on the touch screen display.
 16. The computerizeddevice of claim 11, wherein the first transition and the secondtransition are performed simultaneously.
 17. The computerized device ofclaim 11, wherein the instructions further cause the one or moreprocessors to: after causing the plurality of transitions to occur,determine that a second brightness of the light measured using the oneor more sensors satisfies a first threshold brightness value; andpresent predetermined information on the touch screen display inresponse to determining that the second brightness of the light measuredusing the one or more sensors satisfies the first threshold brightnessvalue.
 18. The computerized device of claim 17, wherein thepredetermined information comprises a clock.
 19. The computerized deviceof claim 18, wherein the predetermined information further comprises analarm indicator.
 20. The computerized device of claim 11, wherein theone or more sensors comprise: a red-green-blue color sensor that isconfigured to detect the color of the light; and an ambient light sensorthat is configured to detect the brightness of the light.